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ABSTRACT 

Noting that present evaluation systems are so limited 
that they are neither reliable nor valid, this monograph critically 
reviews studies designed to evaluate composition programs at four 
major universities. The book offers theoretical and practical 
guidance through discussion of generalities from the four studies and 
pertinent questions and guidance to evaluators of composition 
programs. The first chapter looks at the state of the art of 
evaluating writing programs, discussing the need for such evaluation, 
and at two dominant approaches to writing program evaluation. The 
second chapter discusses a quantitative model of writing program 
evaluation in terms of four university studies, giving an overview of 
the dominant quantitative approach. Chapter 3 discusses a framework 
for evaluating college writing programs, including five components of 
writing program evaluation; and the final chapter discusses 
accommodating context and change in writing program evaluation. 
(HTH) 
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Studies in i 

WRITING V- RHETORIC 

9 

In 198()» the (>)nffrfiut» on (-ollep* (A)mposition and (^oninuniiea- 
tion perceived a need ibr provjdini^ pubiishinji; opportunities for 
monographs that were too lon|i;ibr publication in its journal and too 
short for the typical scholarly biH)ks published by The National 
(louneil of Teachers of Ennlish. The 5>ludies in Writing and Rhetoric 
series was conceived, and a Publications (Committee* established. 

Monographs to be considered for publication may be speculative, 
theoretical, historical, analytical or empirical studies; or other 
works c*<mtributing to a better understanding of composition and 
communication, including interdisciplinary studies or studies in re- 
lated disciplines. The SVVR series will exclude textbooks, unrevised 
dissertations, lHX)k*length manuscripts, course syllabi, lesson plans, 
and collections of previously published material. 

Any teacher-writer interested in submitting a work for pubhca- 
tion in this series should submit either a prospectus and sample 
manuscript or a full manuscript to the NCTE l3irector of Publica- 
tions, 1111 Kenyon Road, Urbana, IL 61801. Accompanied by sample 
manuscript, a prospectus should c*t)ntain a rationale, a definition of 
readership within the CCCXl constituency, comparison with related 
extant publications, a tentative tablt* of contents, an estimate of 
length in double-spaced HVz x U sheets, and the date by which full 
manuscript can be expected. Manuscripts should be in the range of 
100 to 170 typed manuscript pages. 

The present work serves as a model tor future SWR monographs. 

Paul () Dea 

NCTE Director of Publieaticms 
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What do \\v need to find „i,t xvlit'ii wc rvaluatt' a writiiiK proKraiu? 
How ilo wf (k'UTiiiiiif whfUitT a pro«rani is all that it should lu"? 
Oil the lac«> of it. tlif answer sffiiis siiiiplf nu)iiKli. \Vf lu-ftl to know 
whfther stutlfiit writiiiK is iinprovinn. wliftluT stiu'lfiits at tlif (>iid 
of a t'oiirsf (or st-rifs ot c'«)iirst's) art" writinu iM'ttt-r than tlu'V wt-rt- at 
the hfUiiiiiinK. lo tlftfrniiiu" wlu'tlu'r studt'iit writiiifi luis iiiiprovfd. 
what Wf iiffd t«) tlo is fompart' writinu stiulfuts did farly in the pro- 
i^ram with writing tliry did later in the projirain. 

This sort ol" comparison thn-s providf inlbrniatioii of a s«)rt that, at 
oiif tinif iiii|,;ht lia\r sffiiiftl c|iiiU' adi-qiiatf. H«)WfvtT. as wf rt-ad 
Ev4iluati»fi CaUefie WrtfiMg Pronratm. it bt-c'onifs clfar that we may 
no longer assume that evahiatiuK stutleiit writinu the same thing 
as evaluating a comjjosition pn)grain. As Witte and Faigley |M)int 
out, a writing program is much iiu)re than a collection of stutlent 
pa|M'rs. It is a complex set «)f' interrelated activities that exist in an 
even more complex set of sch.)larly. institutional, and interpersonal 
c«)ntexts. To evaluate a writing program, we must answer a number 
of t|nesti«iiis. j)nly one of which pertains to the improvement of stu- 
dent writing. 

Do we, in fact, have a writine. immrum? Many composition pro- 
grams are staHed by a wide range of faculty—part-time instructors, 
gratluate stutleiits. lull-time faculty whose principal scholarly in- 
terest may ox may not be annpositioii. ('oiise(]iieiitly. we neetl to 
ask: "Do these faculty share any coiumoii assuni|iti()iis concerning 
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tlif tcacliiiifi ol cninpositimi? Arr tlinsr assiiiiiptiniis ivllritrcl in 
tlieir assiwiiiinits. rvaluativr priKM^iliirt^s, tcacliiim piiK iMhiirs. and 

tfwe do hare a tvntiufi firofinim, hou stahlr uill it pitn v oin' 
time^ What arr the liircrs— tvu., adiiiiiiistnitivc* dt'i isiims. rliaimrs 
ill studtMit populatiini—tliat arr likt^ly to iiilliiriirc^ lariilty iiunalr 
uiicl/or pcHuniiaiu r? 

/s the program hkely to hat e amj lontl term mfiueiu e on stmleiits' 
UTl7i»«y 1)*) studnits Iravr {\\v prouram witli iiu rrasi'd iniifidrnc<» 
ill tlicir ability? On they find any timniM tiim brtwrni tlir wiitiim 
tliry did lor tlirir rdmpnsitinii touisr and ihv \\ritin« lliry dn Inr 
othvr t nnrsrs? Indrrd. dn tliry actiialK wrilr Inr otlirr cnm si»s nr is 
the I'OiiipiJsition pniuraiii some snrt ijI nasis/aiu)inaly:' 

This list ol unrstinns is lUMtJirr rxhaustiw imr arliitiary. As Wittr 
and Kiiulry make ilrar. we AVe just bruinniim to nndr island tlir 
kinds ol issues writintt prourain rvalnators iinist considrr. To idrn- 
tily tlit'Si' issues, Witte and Kainiey review lour major evaluation 
studies, askinn about iMeli a series ol cpiestioiis tliat apply to evcTy 
aspect ol' thenrx, pi^dauony. ami research- What do we presently 
know? What assumptions are we makiim and how do those assump- 
tions limit our knowledye? Are those limitations necessary or desir- 
able? What do we still need to know? 

By askinu sueb hasie ipu^stiims as these. Witte and Hiiuli^y will 
not make life any easier ll»r prouram evalnatois. \N itte and Kiiuley 
make it clear that we camiot evade certain questiiMis if we want to 
claim we hrtti;i^» lAaluatinl a writinU pcouram Vet even as they help 
us see new questions, they also help ns umlerstand the cimc<»ptual 
Iraniewiirk that makes these rjuestions important and thij|t promises 
to enable us to answer these questions. 

Perhaps ini«v important, Witt(» and KaiuK^y's work lets the Pub- 
lication Committee ol Studies in Writinn and Hheuiric articulate 
one of our main concerns. Alpresent, it could tMsily be aruned that 
many composition prnHrams do not represent a discipline but rather 
are uii ad hoc n^sponse to4be current literacy crisis. To establish 
beyond question the authenticity ol cnnipnsitinii as a discipline. 
\\ v must liave sonu* aurei^inent about tlienry. nu»thndn!ou\. and the 
kinds nf (piestioiis that are worth askinU- We hope that the Witte 
and Fainlc^y text— and. iiidei^d. ail the titles in tlii. series™ will help 
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us work toward that aRretMntMit and, ultimately, toward the estah- 
lishiiUMit of coiii{>osituui as a discipline. 

Tray, Setc York 
January 1981) 
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The State of the Art of 
Evaluating Writing Programs 



I. IJ The Need for Evaluation 

"OCTOBER," MARK TWAIN is reported to have said, "is one 
of the peculiarly dangerous months to speculate in st(K*ks. Others 
are July, January, September, April, November, May, March, June, 
December, August and February." The same could be said about the 
evaluation of college writing programs. Nevertheless, whether by 
choice or circumstance, most composition specialists eventually par- 
ticipate in an evaluation of a writing program, gathering and inter* 
preting pieces of information that will contribute to decisions about 
the program or particular courses in it. From the conception of an 
evaluation, from the planning stage to the final report, the evaluator 
pf college writing programs must confront a variety of difficult ques- 
tions. Among these questions are the following: What purposes will 
the evaluation serve? Stjould the per$on(s) charged with evaluating 
a writing course or program be directly associated with it? What will 
be the basis or bases of judgments about the program? Will teach- 
ers' testimonials bo accepted as evidence of the program s eflfective- 
ness? How will the different evaluation data be relied upon in judg- 
ments about the program? Will student performance or achievement 
be considered the most important source of evaluation data? If mea- 
sures of student performance are deemed acceptable a»> sources of 
evaluation data, how will those measures be administered? Will the 
methods and the procedures of the evaluation affect the normal pat- 
tern of learning? Should the evaluator know the expected outcomes 
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of thv proKram in advaiur oJ the evaluation? Arc (jnantitatixe or 
qualitative methods to he used, or are hoth to he nsc^d? Wliieh will 
he weighted most heavily? Are attitndinal and aHeetive ehanses in 
students as well as eognitive ones to he uieasnred? Are the instrn- 
munts tor doinK so valid and reliahle? Are sufficient funds KoiuK to 
he made availahle to carry out the evaluatitin? 

i*lthouKh evaluations of writing programs are carried out each 
year in many aillcKcs and universities across the country, the litera- 
ture on composition research is with hut lew exceptions silent on 
how to answer cpiestions such as the ones we have posed. This is not 
to say that no lK)dy of scholarship adtlresses evaluation. To the con- 
trary, even a hastily conducteil lihrar)' search will turn up numerous 
hooks and articles addressing the philosophical' and practical- is- 
sues of evaluation. Onl\' recently, howex en have composition spe- 
cialists hcKun to recoKnize the importance and the complexity of 
evaluatiuK an entire writing proKram, to realize that evaluating a 
coUene writing program involves nmch more than assessing the writ- 
ten products of students at the hcKinuiiiK and the ending of a term. 
Major efforts toward improving t^valuation. however are hcin^ made 
hy Richard I^rson s ('onference on College Composition and Com- 
munication Connnittee on TeachiuK and Its Kvaluation in Composi- 
tion/ the Bay Area WritiuK Project.' and the Writing FroKram As- 
sessment Project at The University of Texas at Austin.' Groups such 
as these have hemm to explore some of the many crucial (luestions 
associated with the evaluation of colleKe writinK prosrams. Yet the 
literature on composition research Kives would-he evaluators ver\' 
little to Kuide them through the nia/e of philosophical theoretical, 
and methr)doloKical problems. 

Evaluation research is usually lumpetl toKether with pedagOKical 
research under the more general rubric of applied research. Unlike 
basic research, where the emphasis is on the generation of new 
knowledge for its own sake, applied research strives to put knowl- 
edge to use in practical situations, lu part because of its concern for 
the pragmatic, pedagogical research in general and evaluation re- 
search in particular are sometimes viewed as poor seccmd cousins to 
the important basic research in most disciplines. Kesearch in writ- 
ing is no exception. Some researchers have called for what amoiuits 
to a moratorium on applicu research until more basic (juestions about 
writing have been explored/^ 

In spite of its lack of prestige, applied research in writing is likely to 
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continue. PfdaKOKical research takes as its goal the developniei^t and 
testing of effective composition curricula and instructional ineth<Kls. 
and evahiation research takes as its goal the justification of curricula 
and instruction. In a discipline c )niniitted to the teaching of writ- 
ing, certainh' both goals are iniix)rfant. Yet pedagogical and evalua- 
tion research are, and will cx)ntinue to be, important for two other 
reasons: (I) they can identify significant (juestions for basic research, 
and (2) they can provide some measure of accountability for a profes- 
sion becoming increasingly more accountable to agencies beyond its 
control. 

At the time of the publication of Braddock, Lloyd-Jones, and 
Schoers Research in Written CompositUm in 1963/ research in writ- 
ing was almost exclusively concerned with pedagogy, with how best 
to teach students to write. In the nearly two decades which have 
followed, composition research has diversified considerably, with a 
great body of basic research appearing on discourse theory, compos- 
ing, and writing devek)pment. Often overlooked, however, is how 
much of this basic research has been inspired by pedagogical re- 
search. The most obvious example of how pedagogical research can 
influence basic research in a)mposition is the work of Hohnian and 
VVlecke in the early 1960s.** Their study was designed and executed 
as a pedagogical experiment in which the writing of two treatment 
groups was compared in order to evaluate one pedagogical treat- 
ment against the other Ironically, the Kohman and Wlecke study 
opened up the comiM)sing process as an area of investigation. The 
influence of their study on basic reseaa*h in writing has been ctm- 
siderable, even though their three-stage linear model of compos- 
ing — prewriting, writing, and rewriting — has been dismissed. More 
recently, i>edagogical studies of the effects of sentence annbiniug 
have led to speculations about the cognitive pnK-esses associated 
with writing development and have focused more attention on units 
of discourse above the sentence level. ^ 

Apart from comparing the efletliveness of instructional methods 
and curricula and generating basic research ({uestions, the most 
obvious reason for continuing evaluation research is that pul)lic in- 
stitutions in a democracy are accountable to the public. Account- 
ability of Writing programs has become a much more prominent is- 
sue in the aftermath of the ^'literacy crisis" of the I97ps. Based on 
the results of several national surveys of language abilities — such as 
those conducted under the auspices of the National Assessment of 



ERIC 



16 



4 EvaUiatinu Colleuo WritinK Projiruii^s 



Educational Pronn^ss (NAKP)'"— inaiiy articles Id the popular press 
and many broadcasts ami telecasts during the past decade reported 
that writing abilities are declining at all educational levels. Althounh 
these articles have proBered a lUiinber of* reasons Ibr the decline, 
most echo the conclusion of the widely (juoted Scivsweek article. 
"Why Js)hiniy C:an*t Write"; "the U.S. educational system is spawn- 
ing a generation ol seniiliterates."" 

In response to the "literacy crisis," many collenes and universities 
have inserted new recjuirenients into their curricula and developed 
new programs for teaching writing; and private and federal anen- 
eics have supported several recrnt attempts to make college writing 
instruction more effective. Whether these imiovations are indeed 
improving the writing abilities of'collene students and whether they 
are in fact better than older methods remains wry much in doubt, 
priniarily because neither evaluation theory nor practice has ade- 
cpiately accommodated the disciphne-specific needs of writing pro- 
. uranis and courses. Not to advance discipline-specific theories and 
practices of evaluation may have the efiect of stripping the profes- 
sion of the one means it has for demonstrating; the usefulness of its 
own solutions to the so-called literacy msis. If we as composition 
teachers and researchers do not develop our own models for evalu- 
ating our writing pronrams, others will surely do it for us. 

The purpose of the present monograph is to set out a theoretical 
franiework for evaluating college writing programs. We do not pro- 
pose solutions to the many problems associated with evaluation. 
Rather, we intend this framework to be used to jienerate the kinds 
of (juestions that will lead to comprehensive evaluations. We are 
intentionally pluralistic and eclectic, drawing on many sources in 
constructing this framework. To the extent that our monoj^raph will 
help persons interested in evaluation to understand better the com- 
plex and dynamic nature of collene writing programs, our eflbrts will 
be worthwhile. 

I 2. Two Dominant Approaches to Writhifi Profiram 
Evaluation 

It is curious that so little has been written on the theory of writing 
program evaluation. A va.st htcrature on evaluation exists for other 
disciplines, and issues concerninn evaluation date back as far as col- 
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li'Uo writinu courses thnnsclws. Kwn hvhnv the turn of the trn- 
tury. infort'iKos about tin* cHtrtiviMirss of instruction wrrr niaclo on 
tin* basis of writinu samples J* Anil today, many judunu'uts an* niatU* 
al)out ihv t'HtTtivnu'Ss of writinu prouranis on the l)asis of linntnl 
c'viilcMRe. such as the year-to-year scores on exit e.xauiinations from 
writing courses. Kven thonuh many writinu prourani directors be- 
lieve that sucli measures il<^ not adeciuately reflect the complex mis- 
sion of'writinu prourams, their frustration has brouuht few efforts to 
amend ihe situation. 

If there is no developed body of ther)r\ for writinu program evalua- 
tion, writinu prouranis are nevertheless evaluated with some fre- 
(juency usinu one of two approaches. The first and most common 
approach is the expert'opinion approach. The usual proc^'dnre in 
this approach is for an 'expert" to visit a writinu prouram and then 

\ juil^e its merits. The evaluation depends heavily on the expert *s im- 

prt»ssions tim\ knowledKe. thus niakinu the evaluator central in the 
evidtiation of the prouram. This approach is an extension p{ fion- 
profirammatic evaluations routinely conducted in collcHes and uni- 

\ versities. U\ such nonprugranunatic evaluations, teachers of writhiy 

are called upon to evaluate students, colleauues. courses, textbooks. 
\ curricula, departments, and administrators. 
^ In spite of its widespread use. we refrain f rom callinu the expert- 
^ opinion approach a niodel for evaluation. Kven thouuh the expert- 
opinion appn.ach seems to provide the basis of the (Council for Writ- 
inu Frourani Administrators' forays into evaluafion. we have been 
able to find very few accessible d(K'uments describinu either in- 
stances or rationales for this approach, lu one such dwinnent. the 
WFA Board of (Consultant Kvaluators lists a series of (juestions capa- 
ble of lueneratinu descriptions of niany components of writinu pro- 
Kranis. lIoweviT, the \>'R^ *<)ard never indicates how the residtant 
descriptions shotdd fipir * \ jrouraui evaluations. The WFA Board 
apparently sees no distinction between description and evahiation: 
while it oHers its set of (jueMions as "a too! that WPAs and their col- • 
lc»a)ijui»s niay find useful i^ reviewinu their prourams* f^oals. needs, 
and procinlures, " it pnvides no uuidance for arriving at evaluative 
judgments. 

In practice, the expert-opinion approach xaries a great deal fron. 
evaluator to evaluator. from site to site. In some locations evaluators 
collect descriptions — in the form of course descriptions and syllabi 
— of the program being evaluated; in others they do not. in some 
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locations evaliiators talk to tiMcluTs and stiulnits: in othrrs tlu*y ilo 
not. In sonif locations cvaluators x isit class<»s: in others they do not. 
hi sonic locations cvalnators examine student papers; in others they 
do not. Thf'only yhinnion denominator seems to lie the presence ol* 
an "expert," the definition of which is as obscure as the approach 
itsfH. 

11 wc cannot construct a model tor the expert-upiniO!i approach, 
we can identify some of'its imderlyinn assumptions. First, as CJard- 
ner and Houi^e both urfte,'^ the principal assumption is that the best 
evaluator is an expert on the thin^ or actixity bein^ evaluated. 
This assumption, however valid it may appear on the surface, is 
open to question. In the evaluation ot writinn proHnwns, a Rood 
deal hangs on one s identification of "expert." is an expert evaluator 
of a writinK program someone who is acknowledged as an effective 
teacher of writing? Is an expert someone who has sucwssfidly di- 
n*cted a writing program in another setting? Is an expert someone 
who is a member of this or that professional group? Is an expert some- 
one who has written textb(H)ks for college wnting classes? Is an ex- 
pert someone who understands and contributes regularly to the de- 
velopment of new knowledge in the field? Is an expert someone who 
esp4)iises a particular theory of writing, or curriculum, or in.struc- 
tioii? The inherent weakness of this underlying assumption is that 
very few experts tend to be knowledgeable in all areas of writing 
programs. 

A second assumption, namely, that the criteria upon which judg- 
ments are made are appropriate, derives from the first assumption. 
This sec*ond assumption runs contrary to the nature of expertise since 
expert judgments are likely to be colored by the particular area of 
expertise. Someone who is an expert in organi/nitional theory may 
be able to make valid judgments about the cBectiveness of the ad- 
ministrative structure of the writing program but be incapable of 
making valid judgments about the appmpriateness of a specific cur- 
riciilar cxjuiponent. Similarly, a textbook writer may be a responsi- 
ble judge of the organization and presentation of text materials but 
not have the experti.se recjuired to make judgments about the suit- 
ability of the text materials to the institution's goals. In all possible 
approaches to writing program evaluation, but especially in the 
expert-opinion approach, Kenneth Burke s reminder that a way of 
seeing is also a way of not seeing seems apropi^s. 
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A third assumption ol tlie rxpiTt-opiiiioii approudi (*iiiaiiat( s Iroin 
the firs^two: that the expert or experts seleeted to evahiate the pro- 
Kraiii are capable of proeessiiin all of the relevant evaluation data in 
order to make jndfiments based on those data. There iscoiisideralile 
€»vidence that experts whc* share the same haclcKrouiid knowledge 
combine data in different ways to arrive at sometimes disparate jiidu- 
nuMit?i."' In addition, expert-upiniun evaluations rarely ackiiowiedue 
principles of sample selections or neiierali/abihty when considering 
evidence, nor do they make use ot'cpialitative methmls of evalua- 
tions because "experts ' in writiiiK seldom are trained in these areas. 

As a consequence ot these assumptions, the expert-opinion ap- 
proach to writing program evaluation is larnely atlieoretical making 
any evaluation model impossible to extrapolate. In some instances, 
experts have perhaps niven sound evaluations of writing programs. 
But V experts have also been used to make proprams appear better 
tiian they are or to delay unfavorable administrative decisions. Since 
the results of expert-opinion evaluations ustially are not published 
or presented at professional mectiii^s. there have been no advances 
in this approach o\ t the years. All is dependent upon the knowl- 
edge, biases, commitment, and sensitivity of the evaluator. 

The second dominant approach— a quantitative one which typi- 
cally uses pretest-i)osttest designs— also lacksan articulated theoreti- 
cal basis. The (juantitative approach, however, differs significantly 
from the expert-opinion approach in several ways. The evaluator is 
not at the center of the evaluation. Instead of relying on subjective 
impressions, an evaluator seeks objectivity in quantitative measures 
which can be used across evaluation settiims. Quantitative evalua- 
tions are f'recpieiitly publishecL allowing other researchers to exam- 
ine their assumptions, methodoloyies, and n'sults. Kesearchers have 
attempted to learn from other researchers in order to refine their 
research designs and procedures. 

While there is no articulated model Ibr conducting (piantitative 
evaluaticms of writing prourains, we believe such a nicKiel exists and 
has informed some major pretest-posttest studies of writing pro- 
grams during the past two decades. In the next chapter, we extrapo- 
late that model. 
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The Quantitative Model of 
Writing Program Evaluation 

IN CHAPTER 1 WE akgued that thehe have been two ik)mi- 
nant approaches to writrng program evaluation — the expert -opinion 
approach, which is an ad hoc response to the need for evaluation, 
and the prptest-posttest approach, which' is based on an unarticu- 
lated model for writing program evaluation. In this chapter, we will 
examine that unarticulated model by revier' ^ng tour major pretest- 
posttest evaluation studies: the University of Northern Iowa study, 
the University of California San Diego study, the Miami University ^ 
study« and the University of Texas study. We might have chosen 
other studies as well, but these four serve to raise the most im|K)r- 
tant issues evaluators must facv. The four studies tbcus in different 
ways on two related questions: Does college writing insrtruction 
positively aflfect the development of writing abilities? and Is one 
type of c*omposition instruction more effective than another? To- 
gether these studies point to the need to acc*ommodate the large 
numl)er of curricular, instructional, and c*ontextual variables of col- 
lege writing programs in au evaluation design. 

IL 1. The Univemty of Northern Iowa Study 

Most evaluations of college writing instruction have addressed 
the issue of whether one kind of writing instruction is more effective 
than another. One exception is the study conducted at the Univer- 
sity of Northern Iowa by Ross M. Jewell, John Cowley, and Gordon 
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Rhitm.' The Northern Iowa study sounhi an answer to the hroader 
question. Does any kind of u)llege writinn instniction make a dif- 
ference in the writing ahihties of «)lleKe students? We ehose to ex- 
amine this study l>eeause it illustrates two problems that fretjuently 
appear in evaluations of writiiiK programs and courses: (I) the fail- 
ure to understand and am)minodate differences between com|>osi- 
tion courses, and (2) the failure to recoKiiize and control diflerences 
between noncom|K)sition courses of study. 

II 1. I. The Design and Results of the Northern Iowa Study 

The Northern Iowa study paired 2,()80 freshmen enrolled ii five 
diflerent universities in the fall of 1964. The paired students were 
matched on the basis of sex, ACT or SAT smre, a theme scop:, and a 
sti)re indicative of ccmibined p<:Tlbrmance on the C<M)perative En- 
Klish Tests. Enplisb Expression (C:00F) and the ColleKe Entrance 
Examination Board English Coini)osition Test (CEEB). One niem- 
Iht of the I,()40 pairs t(M)k «)urses in writiiiK and in other academic 
subjects while the other niemlwr ttnik c^ourses in other subjects only. 

Jewell, Ct)wley, and Rhuin report that three tests— the COOP, 
the (JEEB, anr? a theme — were administered to the students in the 
study at four different times, once near the beKinning of their col- 
lege careers and once at the end of the first, second, and finirth se- 
mesters of study The COOP and the CEEB. which are both objec- 
tive tests, were scored according to the publishers' guidelines; and 
the themes were rated by Educational Testing Service personnel 
using the prixedures then employed by ETS.' The analyses per- 
formed on the Northern Iowa data generally indicated that at the 
end of the first two semesters, the students who were enrolled in 
et)inposition performed better than their paired counterparts who 
were not. However, by the end of the fourth semester, students 
who had received no formal instruction in college-level writing per- 
formed as well as those students who had received writing instruc- 
tion as freshmen. Jewell and his tiilleagues thus concluded that 
freshman composition teaches writing skills that develop naturally, 
but at some later p<)int in the student s academic career. 

II. 1. 2. A Critique of the Northern Iowa Study 

The Northern Iowa study has suggested to many that teaching 
courses in freshman writing is pointless, a conclusion which Jewell, 
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Cow\c\\ and Hhniii did not tluMnsHves draw. Vet tlu» study does snu- 
Kcst that conrsfs in freshman nnnpositicni may he of limited value. 
But even that tentative nniclusion may hv (|nestioned. Kiven tlie 
design uf the study and some ol tlie assumptiims wliich underhe the 
design. 

Althcnigh impressively complex and and)itious, the Northern 
Iowa study has a numher ot flaws in its ilesign. These Haws resulted 
primarily from two assumptions the Northern hiwa investigators 
made prior to cH)llectinK the data on which their analyses were 
hased. Both assumptions concerned the nature of the writing pn)- 
grams at the five participating universities. First, the investigators 
failed to acknowledge in their research design diHerences anunig 
the five programs, even though their descriptions (see pp. 23-^24) ol 
the five eoni)K)sition programs indicate that considerahle variation 
among the programs (»xisted, \wi\\ in terms of 'content" (their term) 
and in terms of the amount and kinds of writing the students did. 
Indeed, the '*cmn|)ositicm program ' at one participating univer- 
sity— The University of h)wa— was quite unlike any of the other 
programs. Second, tlie investigators did not attend to diflerenees in 
instructional methcKls used in the various programs or hy individual 
teachers within a particular composition program. These two as- 
sumptions led the Northern Iowa investigators to ignore diflerenees 
in eurricuhnn. student populations, program and cmirse goals oroh- 
jectives. and instrueticmal nietlimls. 

The Northern Iowa evaluation also assunu'd that all ncnie<nnposi- 
tion instruction which either group of students received was un- 
diB'erentiatcd. l)oth within and across the five instituticms. That is to 
say. the investigators assumed that taking a course of study in chem- 
istry att'ects the development of writing ahilities noditterently than 
taking a course of study in art history or literature. UowevcT. an al- 
ternative assumption— that difl'erent etnirses of study had difl'erent 
eHects on students' writing abilities— is perliaps the nuire plausihie 
one. For example, students enrolled in an engineering degree pro- 
gram prohahly did considerably less reading and less writing during 
their first two academic years than did students enrolled in a pro- 
gram leading to a degree in history Jewell. Cowley, and Rlunn re- 
mained silent alM)ut three important matters embedded in their sec- 
ond assumption: the nature of the degree programs in which the 
students were enrolled, how these degree programs may liavi* dib 
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lercd 111 proniotiiiK laiiKiiaKc* skills, iitiil tlu* (listriliiitinii oriiidividiiiil 
subjects aiuloi pair(*(l siihjc^ts in t\\v diHernit niiirsc^sot study within 
and across the five uiiiv(*rsitics. 

The invest inators' (uilure to explore these assumptions and their 
failure to control the variables these assuuiptioiis ijjnore make' 
the resuhs of their comparisons dillicult to iiiteqjret. It the I. (MO 
matched pairs formed at the beuiimiuK oCtlie study hud ren.aiiu d in 
the study for the bill two years» these assumptions would be less 
problematic than they are. and the iiiHuence of particular instruc- 
tional proKrains. whether in coiii|)osition or in other disciplines, on 
the results ininht have been less. By the end of the first semester 
however, the 1,(M0 pairs were* reduced thnnmh attrition to 597 pairs: 
by the end of the second seinesten to 3f>5 pairs; and by the end of 
the fourth semester, to 122 pairs, sliKhtly more than 10 percent 
of the oriKinal sample. If the five coin|>ositioii proKraiiis had been 
identical in all respects (but they were not) and if the iioncomposi- 
timi courses of study had been id(*ntical (but they probably were 
not) Ibr the reinaininK 122 pairs, we could place more confidenei' in 
the results of the study However, the Northern Iowa investiKators 
did not re|K)rt the distribution of the 122 pairs either within coiii- 
IMisittoii proKrams or within iioncoiiipositioii courses of study in any 
of the five universities. 

The problems which result from iuiioriiiK such variables are many 
For example, if a substantial iiumherof the 122 students enrolled in 
freshman ciimpositioii courses were enrolled in a particular loni- 
|M)sitioii proKrani, any differences between the two groups* perfor- 
mances at the end of the fourth semester miuht be attributed to the 
ineffectiveness of one particular composition pro» ram anioiiu the 
fiv e used, rather than to the iiieHectiveiiess of freshman composi- 
tion in general. Attention to curricula^ instructional, and c<mtex- 
tual variables is mandatory in well-designed evaluation research. 

The two assumptions we have addressed— that all com|X)sitioii 
courses are equivalentand that all noncom|)ositioiKH)ursesare equiva- 
lent — illustrate the need for careful attention to complex seti of in- 
teractinn variables, whether they be contextual, curricular, or in- 
structional in nature. Altbouj^i we have criticized the design of the 
Northern Iowa study, we have not condemned the study itself, for 
the investigators present virtually all of their findings cautiously. The 
Northern Iowa study is one of the most ambitious and most complex 
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evahiation studies onVrshiuaii composition rvt»r attCMuptrd. lii lact, 
its efiiiipli'xity is oiu» reason the stutly was selected lor rvvivw Uvrv, 
Thosf who undertake sucli uniiplex endeavors deserve notliinn l)nt 
our respect, lor tlie difficulties tliev encounter are uian\. 

11. 2. The Vnivemttj of California San DiVfio Study 

While the Northern hiwa study sought to determine whether any 
kind ol' writinK prourani increases writiim ability, luost evaluation 
studies have l)een primarily conceriu'd with Hndiim answers to the 
second (piestitm we i>osed earlier: Is one method ol teachinu IVesh- 
man ct)iupositi(m more eflective than another? Several studies have 
addr(>ssed the (jucstioii directly, but one study addressed it in- 
directly. That study is the one recently completed under the direc- 
tion ofDonaUl Weslinu at the University of Clalifornia Sau Dieno/ 
The differences between the "prouraius ' ex aluated in the San 
DieHo study are very much like the differences l)etween "ecjuiv- 
alent" courses within the same prouram. We selected the San Dieuo 
study lor review here because (1) unlike the Northern h)wa study, it 
attempted to acccmuuodate diHerences in the wax* writinu is tauuht, 
(2) it relied on more than one measure of writintt course or prouram 
effc'Ctiveness, (3) it illustrates some ol the difficulties assin iated w ith 
inierinu c<mrse or prouram ('ffixlivencss I'roni writing samples, and 
(4) it illustrates the relative nature ol' writing program evaluation. 

II. 2. 1. The Design and Hesults ol'the San Diego Study 

The subjects used in the Sau Diego study were 175 Ireshmen un- 
evenly distributed across the lour colleges within the Vniversitv. 
Kach ol* the lour groups— which ranged in si/e Irom 35 to 50 stn- 
dents— produced three essays which were submitted to holistic 
evalnutioi> on a nine-point scale by two independent raters, with a 
third rater used to reconcile large differences between tlie scores 
assigned by the first two raters. Kach ol' ihv lour groups received 
instruction in writing, but that instruction differed considerably 
across groups. For exaniple\ while two ol'the colleges* comiH)sition 
programs were tied to tlie contents ol' other disciplint^s, the other 
two were not. One ol'the programs tied to aiu)ther content required 
students to write on siktA issues, deriving topics from a reader while 
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till' otIicT rcM|iiirc'il its stndc^iils to write (hi topic s Iroiii tlif li(iiiiaiii< 
tics. All four uronps ol stiuhMits wvrv reported to liavr simiificaiitly 
iiicToaseil thrir avrrujif holistic scones Innii tlir first to tlir srcoiul 
essay and Iroiii thr first to tlir third essay. 

Soiiu' ol thr stiidiMits also took thr KiiKlish AchirvcMiiciit lest 
(EAT) at two diltrrrnt tiiiirs. oiur prior to udiiiissioii and oiitr alter 
c'oinpietiiiu a course in eolleue writiiiu. This pretest posttest use ol 
the KAT was adopted in order in determine whether writinu in- 
struction ill the fiiur colleijes "led to improvement in the editinu 
cimipoiient of writinu ' (p In achlition. the KAT was admin- 
istered to a ^roup ol students who had not heen exposed to writiuu 
instruction in any of the linir colleues. Analyses ol the KAT results 
indicated that none of the five uroups— the lour unuips drawn Innn 
tlu» four coih'Ues nor the "control" «ronp— realized statistically siy. 
nificant ^ains on the KA 1' 

In additioit to the use ol three essays as measures ol the students* 
writing abilities and the two administratfons ol tlie K AT as measures 
ol" editinu skills." the San Dieuo study employed a nmnlier of other 
evaluative t(H»ls, For example. James MoHett ' serM^d as an outside" 
evahiator wlio visited classes, talked with teacluTs and administra- 
tors, and solicited reactions to the liuir projirains rroin students. The 
investigators also collected reactions Iroin students to the composi- 
tion pro«ranis in which they were enrolled. For this purpose, the 
investiuators used a 37-item course and prolessor evaluation ques- 
tionnaire, a l<Kally designed Likert-type instrument which appar- 
ently had heen usc»d at San Die«o lor several years. In addition. '):) 
students were interviewed In persons involved in the sludy in an 
attempt to arrive at some "confirmation of impressions uenerated 
tnim other sources ofilata (p. 45K 

11. 2. 2. (Jrilicjue ol the San Die^o Study 

The report of the San Die«o evaluation concludes that in ueiieral 
the four iinder«raduate colleges at I'C San I)ie«o did e<|ually etlec- 
tive johs in improviim students' writinU- 

The study is not. however. Iree ol prohlems in research desiun 
and meth(Klolo«y. some ol whicli are noted hy tlie investiuators. ' 
These prohlems in design and iiiethodo^oKy need to he placed in 
perspective, since tin* San Die^o evaluation was an evahiation (juite 
difl'erent Irom the Northern Iowa study. Whereas the Northern Iowa 
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evaluation sought, in efl'ect, to determine the relative efficacy of two 
diflferent treatments — one involving composition instruction and 
one not — the purpose of the San Diego study was "to try to evahiate 
programmatic strengths and weaknesses of each college program in 
context rather than in competition" (p. 34). Thus overt comparisons 
of the writing programs of the four colleges were prestmiahly be- 
yond the scope of the San Diegtj evahiation; however, Wesling and 
his colleagues structured the report and evaluated the writing sam- 
ples in such a way that implicit, if not explicit, c^onipurisons across 
the four tx)lleges were made throughout. 

The features of the San Diego study which make comparisons 
across the four programs difficult can he illustrated very easily. If, 
for example, the investigators had been serious about evaluating 
each of the four programs "in amtext rather than in ccmipetition,** 
then it would have made more sense to write a sei)arate report for 
each program. Instead, the investigators chose to write a single le- 
port in which, among other things, they put in the same table the 
results for the four colleges and employ such expressions as "cross- 
college comparison/' "Warren, unlike the other colleges,** and "high- 
est in Third College." Throughout the report, the reader is invited 
to make the very comparisons the investigators say they intended to 
avoid. 

These comparisons were, in fact, unavoidal)k*, given the proce- 
dures used for conducting the holistic evahiations. The investiga- 
tors rec^phized that the four ctillege writing programs differed in 
importafit ways from one another. One of the ways in which they 
differed was in the kinds of writing the students in the various col- 
leges were asked to do, a matter documented by the investigators 
themselves. Indeed, the researchers included the following sen- 
tence in their report: "Among the variables we could not control lor: 
discourse mode, the amount of time elapsed between essays, the 
number of hours of instruction, different forms of instruction, difler- 
ing amounts of collateral reading, differing ways of generating topics 
for writing * (pp. 34- 35). One might be inclined to think that the 
idck of control over these variables would be inconsequential in an 
evaluation whose purpose was not to compare four composition 
programs but rather to evaluate each program in its own context. 
However, the essays — which were written on different topics, in 
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different modes, and at different times by the lour uroiips— were 
pooled for the hoHstic evahiations. Thus the raters knew the topic of 
an es ly if it was "indicated . . . by the title" (if there was one) or 
"by whatever the reader . . (could) infer from the thrust of the 
Riven piece of writinR" (p. 28). 

In holistic scorinK. judnments of writing (juality are always rela- 
tive. Haters give a particular score to a particular paper in relation 
to the scores assigned t«) the other papers in the set. A holistic train- 
ing session mi^ht be defined as the process b\ which experienced 
raters of student writing are forced throuKh ^nnip pressure to aban- 
don their own ideas of writing quality anti to ad«)pt others which are 
relative to the rating group's view «)f writing (|ua!ity. relative t«) the 
set of essays being rated, and relative to the need to distribitte es- 
says acnws all scoring categ«)ries. Whenever ratings— whether holis- 
tic orot|a'rwist — are made relative to all the papi-rs in the set. all 
ratingSare based on explicit and implicit comparisons among the pa- 
pers in the set. 

W ith the papers from the four colleges pooled, the raters in the 
San Diego study had no choice but to compare the writing of stu- 
dents in one c«)llege with the writing «)f students in other colleges to 
determine the merit of one essay in relation to all other essays in the 
set regardless of the topic on which the essays were written, the 
rhet«)rical puqiose for which they were written, and the nn)des in 
which they were written. In short, the procedures employed man- 
dated comparing essays across colleges. These wry comparisons, 
however, were the ones the investigators iiulicatecl could not Ik- 
made because the four writing programs were fundamentally dif- 
ferent. Thus while the San Diego investigators sought to i voiil oiie 
of the problems we identified in the Northern Iowa stud* . i; did 
not do so. 

Although we have been criticial of certain features of tlie San 
Diego evaluati«)n study, we believe that Wesling and bis colleagues 
are to be commended for recognizing that use of a single source of 
data will n«)t result in a g(M)d writing program evaluation. Because of 
this recogniti«)n. t)ie San Diego investigators collected and anab /ed 
different kinds of data fn)m a number of diff-rent sources— objec- 
tive measures of writing ability, writing samples, measures «)f stu- 
dent attitudes towards courses and instructors, personal observa- 
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tions and interviews by an "outside" expert, and innlepth interviews 
of students. No single set of data provided the basis for the positive 
evahiation accorded the four writing programs; rather, the inves- 
tigators used all available data as the basis Ibr their evaluation of the 
four programs. 

Because no generally acceptable pnK-edures and materials for 
evaluating writing programs have been developed, we have limited 
our criticism of the San Diego study to aspects of the holistic evalua- 
tion of the student papers. Although not all of the issues of relia- 
bility and validity have been addressed, procedures for evaluating 
student texts hoiistically have been established and the nature of 
such evaluation has been defined/*thus justifying our criticism. Our 
criticism of the particular method of rating the San Diego papers 
hoiistically raises some important (juestions about the validity o! 
that procedure, for a meth<Ml can be valid for a particular puq^ose 
only if it is consistent with that purpose. The lesson to be learned 
from the San Diego study is that writing program evaluators must 
develop and use research procedures ct)nsistent with the purpose of 
the evaluation. 



11. 3. The Miami University Study 

Some studies have attempted to avoid the problem illustrated by 
the San Diego study by comparing diflerent approaches to teaching 
college writing. These studies try to control, through both research 
designs and statistical procedures, the major curricular and instruc- 
tional variables. However, these variables are very difficult to con- 
trol, as illustrated by the Miami University sentence-combining 
study conducted by Andrew Kerek, Donald Daiker, and Max Moren- 
berg. The purpose of the Miar » study was to determine the ettec- 
tiveness of a "sentence-combining curriculum" in t»omparison to a 
"traditional" composition course.^ The Miami study was selected for 
review here because it (1) illustrates the difficulties asscwiated with 
the failure to separate curricular and instructional variables in- 
volved in comparative evah^ations of writing courses, (2) illustrates 
the problems of defining both curricula and instruction in writing, 
and (3) illustrates the necessity of controlling instructional and cur- 
ricular variables through carefully conceived research designs. 

1 
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IL 3. 1. The DesiKn and Results of the Miami Study 

In the Miami study, 29() be};innin}{ tH)lleKe freshmen were used to 
form two groups c^onsistin^ of six elasses each, One uroup was called a 
"contror* Kroup and the other was called an '*experinientar' nn>"p. 
even though both groups underwent writinn instruction of some 
kind.** The "experimentar* Kruup, consisting of 151 sul)jects, was ex- 
posed to what the Miami investigators called "an exclusive sentence- 
cmibininK curriculum,"*' while the other, consisting of 139 subjects, 
was exposed to "tradititmar'"* instructitm in writinU- Pretest and 
posttest essays were collected from all students. These essays were 
written on an informative or expositi)ry topic that allowed students 
to draw narrative and descriptive details from personal experience. 
These essays were collected from eacli nro»P at the be^innin^ and 
ending of a semester of siudy. In addition, the investigators also col- 
lected pre- and posttest reading scores. When the reading scores 
were tH)mpared» neither group was found to lia\ e improved in read- 
ing skills more than the other. However, wlien the holistic and ana- 
lytic scores assign(*d to the writing samples were compared, the 
investigators found that the "experimental" group had realized 
significantly larger gains in writing ijuality than had tlie "control" 
group. So too with comparisons of the writing samples along certain 
dimensions of syntactic fluency. " 

A folloWHip study Wiis also conducted 28 montiis after the original 
data had been collected. Of the original 290 students, 140 partici- 
pated in the follow-up, 6.5 from the "wintrol" gnnip and 75 from the 
^'experimental" group. The fbllow-up was tt)nducted in order "to in- 
vestigate the long-range effects of intensive sentence*combining 
practice on the writing ability of college students" (p. 1130). The 140 
students in the follow-up study were volunteers who were paid to, 
write one essay on the same topic they had written for tlie posttest 
essay in the original study. These 140 new essa> s were pooled with 
those essays originally written as posttest essays by the same stu- 
dents. This set of 280 essays was then rated holistically and analyti- 
cally by ETS raters who were reported not to have known the source 
of the essays they were asked to rate. AnaU ses of tht? fbllow-up rat- 
ings revealed th;'t "on both the holistic and tlie analytic ratings, the 
control-experimental diflerenc^s were statistically not significant" 
(p. 1133), that the "sentence-t*ombining students were no longer su- 
perior to the amtrol students in writing quality" (p. 1134). 
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/ The Miami iiiwstigators initially speculated that the ({nalitative 
increases on the experimental students* posttest essays were strongK* 
linked to the (jiiantitative gains in Hunt s syntactic indices, since the 
group taught by sentence combining increased significantly in both 
holistic scores and s\ ntactic measures while the traditionally taught 
gnmp did not.*^ Other researchers who examined the relationship 
between syntactic maturity and judgments of quality, howevt^r, 
found Hunts indices to be niininuimly valuable for explaining vari- 
ance in holistic sources.'* When the Miami investigators analyzed 
the amount of the \ ariance in holistic ratings of experimental post- 
tests which could be accounted ibr b\' Hunt s indices, they found 
that the sy ntactic factors together '"predicted not quite \^/c of the 
. . . variance" (p. H26). Thus the raters were not inHuenced to any 
great degree by the increased complexity in the sentence com- 
biners* posttest essays. 

H. 3. 2. Criticpieof the Miami Study 

The Miami investigators encountered several problems in defin- 
ing the curricnlar and instructional variables in their study, a fact 
which finds illustration in the differing accounts of the study. These 
problems ■ ^mii fn)in the investigators* failure to determine prior to 
the study itself whether they wx're to examine a curriculum, and in- 
structional method, or both. The 1980 and most recent version of 
tfie Miami study «)tiers poni hoc explanations of why sentence com- 
bining worked and attt^nipts to answer some of the criticisms and 
({uestions raised about the study as originally reported. For exam- 
ple, Mellon and Kinneavy pointed out that the whole-discourse 
exercises used in the Miami study diflered substantially from tfie 
sentence-combining exercises used by Mellon and O'Hare in earlier 
studies. The Miami investigators did not address this issue in their 
1978 reports; but in the 1980 report, they write that the **experi- 
mertal students were exposed to rhetohcalltj based [italics theirs] 
sentence c*ombining, ' that "the sentence-eond)ining method used 
in the experimental sectitms was deliberateb* and unabashedly rhe- 
torical in character" (p. 1099). Klsewhere in the report, the inves- 
tigators say that *'the term sentence combining' is not a wholl/ 
accurate description of the classroom procedures used in the experi- 
mental sections: sentence-combining practice obviousK* went far 
beyond isolated, sentence-level granmiatical transformaticms" (p. 
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1100). CJlearly. the Miami iiivostiKators lacod a difficult i)n)!)ltMn 
in definition, a |)n)l)leni which turned on not only the activity oi 
conibininK **kerner* sentences into lonKer ones and the kinds of 
sentence-amihiniuK exercises used hut also the classnM)ni pnKc- 
diires the teachers of the experimental classes used to lead their stu- 
dents to an understanding of the rhetorical principles they wished 
toeniphasi/e. Ol)Viously, the Miami investigators had expanded the 
notion of sentence combining to include the notitm of discourse 
ctmibining and discourse generation as well. 

The Miami investigators* expansion of the notion of sentence com- 
bining as inherited from Mellon'^ and O'Hare'** was nothing short of 
inspired. Yet because this expansion was not carefully defined prior 
to the study itself, it created a serious design problem for the inves- 
tigators, as reflected in their inability to distinguish between sen- 
tence cM)mbining as a "teaching method" (p. 1116) and sentence com- 
bining as a cnrricuhnn: "The idea of an exclusive sentence-combining 
curriculum had not l)een tested before" (p, 1101). 

If we understand curriculum in its usual sense of "content ' and if 
we understand teaching method or instructional vwthail in its usual 
sense of the means and ways used to teach curriculum, then the ma- 
jor problem in the Miami study becomes obvious. Sentence combin- 
ing is not a writing curriculum; rather it is an instructional methml 

The instructional method of combining short sentences into one 
or more sentences contains no inherent assumptions about the re- 
sulting sentences. That is to say, as originally conceived, sentence 
unnbining in and of itself did not teach a c^mtent. did not address 
rhetorical ct)nc»erns; it taught students how to combine sentences in- 
dependently of rhetorical situations. MelKm, for example, used sen- 
tence ^'ombining to accelerate naturally (xxiirring syntactic devel- 
opment, not to teach the use of rhetorical principles. But the Miami 
study employed sentence combining in a very diflerent way. and for 
an additional reason. In the following quotation, which illustrates 
the instructional materials and methods growing out of the Miami 
study, the Miami investigators use sentence c*f)mhining to teach 
rhetorical principles. Here they discuss three students' solutions to 
a sentence-combining problem: "All three student writers appar- 
ently realize that repeating key terms or sentence elements, es- 
pecially at the conclusion of an essay, is an effective means of em- 
phasis. But all have problems making repetition work. In the first 
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version, the traKinent is more disruptive than emphatic."^' This ad- 
aptation uf sentence a)ml)ininK to teach rhetorical principles con- 
tained in traditional writing curricula is an inipf)rtant iiuiovation. 
one which makes sentence c^omhining an instruct icmal method ap- 
propriate for college writing classes. But the rhetorical principles 
themselves are matters of curriculum, not the method or methods 
used to teach it.'*" In short, the Miami investigators Wv're iiiialiiv to 
decide whether they were testing tor the relativr eB'ectiveness of a 
curriculum or an instructional method or both, (x)nsequently. they 
failed to cx>ntrul some important variables, both instructional and 
curricular, in their study. 

Among the variables left unccmtrolled in the Miami study were the 
amount of writing the students in the t\vo groups did. the amount of 
collateral reading they did, the kinds of writing they did. the rhe- 
torical principles they were taught, the textb(M)ks they used, the 
amount and kinds of homework assignments they did, and the appar- 
ently diverse set of instructional methods used in the two classes. 
These are essentially some of the same variables which were left un- 
c*tmtn)lled in the Northern Iowa study and in the San Diego study. 
With this many variables lelt uncontn)lled, it is — contrary to the 
claims of the Miami investigators — imi>ossible to tell whether the 
**experimentar' classes promoted grc-ater growth in syntactic Hu- 
ency and writing tpiahty or whether the * controK' classes inhibited 
such growth. If one adopts the |K)sition of the Miami investigators 
that tlie ''experimental" classes promoted such growth, determining 
which instructional or curricular variable(s) caused the growth is 
similarly impossible. 

Hie investigators explain their failure to control one of these vari- 
ables in the following way: 

The experimental students did more out-of-class writing than their 
control counterparts. Since all sentence-combining exercises recpiire stu* 
dents to write, experimental students spent much of their homework 
time writing, just as control students spent nnich of their time reading. 
After all. homework assignments that retiuire writing are as integral to a 
seittence-eomhining program as those that re()uire reading are to a pro- 
gram with an essay reader and a standard rhetoric. It follows that ecjualiz- 
mg the total amount of writing hy the control and experimental students 
would have meant changing the hmdamental nature of at least one of the 
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t\v.i ciirriiiiliiins. Hut siitli tontrol iimv haw I»ccmi as iinnccrssarx as it 
was iimli'sirahli'. iH'caiisf ri'sfartli snKj-csts that lu'itlu'i thf aiiiiiunt nor 
till' Iri'qiiciuy ol writiiiK in itsfll iniprtivi-s writintf (jualitv. (p. 1102) 

While it is trui- that soiiu' ri-sfarch— rt'sfarch dd hiuh scIkmiI writ- 
m-— sumji'sts that "iifithi-r thf ainoiiiit nor th«' IrnniciKV of writ- 

iiiK in itst'H improvi's writiim (,iiality." tht- aim t and tht- Jn'.niciK v 

of "writing ill itsi'H " is not what is at issiit- in thi- Miami stiitlv. What is 
at issiii' is thf variahihty in thi- amount and lrt'i|m'iR V of writiim in 
conjunction with thf variahihty in instrnttional nu-tiiods. kinds of 
writiiiK. amount and lri'.|iit'nc> of ri-adiim. typi's ol'tfxt mati-rials. 
and so(orth;and nnU-ssall siit h variahh-s an- controlled, it is virttuilly 
inipdssibii' to attrilMitf causality to one variahic Instead ol another. 
What the Miami study aUows ns to cont hide is that the students en- 
rolled in the writing classes whieh used senteiite-eonihininK I'xer- 
eises iiutpertormed stiuh-nts enroihnl in a course which did not use 
those exercises. Whether the sentellc•e-c•olnl)i||il|^ exercises chd. in 
Jact, increase perlorinance is sinipK not known. 



II. 4. The University of Texas Study 

AhhouKh diHerent in hoth scope and kind from the other three 
studies, the University ol Texas study coiuhicted hy the present au- 
thors complements certain aspects of the earlier three. Like all of 
the three previous studies, the Texas study attempted to measure 
improvement in the writinu ahilities of c-olleue freshmen across time. 
Like the other three studies, it examined the perlbrmances «)f stu- 
dents enrolled in a "traditional" freshman writinu course. Like the 
San Dit'Kti study and the Nortiu rn hiwa study, it examined existinu 
courses of study, rather than creating experiiiiental ones. liike the 
Miaun study, it compared the perJoriiiance of students in a "tradi- 
tional" c-onrse with that of stuth-nts enrolled in a course predicated 
on sentence-expansion exercises. 

The Texas study also diHered in important ways from the other 
studies we have ihscussed. The Texas study, unlike the Miami study, 
thtl not hypothesize the greater efficacy of an ■experimental" course 
over a "traditional " one. Neither c-ouiv in the Texas study was con- 
sidered experimental; hoth were courses which had heen reKiilarly 



ERIC 



34 



22 EyahiatinR C^tillfgf VVriliiiK FroKratns 



tauRht for sewral years. Uiiliko thv Miami study, hourvon tho Texas 
study found signifirant iniprovfiiient in the writing and reading 
skills of students enrolled in "traditional * freshman writing classes. 
Unlike the other studies, the Texas study examined changes in writ- 
ing ability across two different types of writing, not just one. 

But the most imix>rtant difl'erence is that the Texas study was 
based on a research design and analytic procedures that addressed 
the diflferences in instructional methmls, curriculum, and instruc- 
tional media in the courses mmpared. Our puqiose in discussing 
the Texas study in some detail here is to illustrate the extreme diffi- 
culty in controlling major variables when two very different c*ourses 
are compared. 

II. 4. 1. Design and Kesults of the Texas Study 

The Texas study t*ompared the effectiveness of two options for the 
introductory freshman composition course, options which had been 
in place for several years. The two options for accjuiring credit in the 
cinirse diflered with respect to instructional setting, method of in- 
struction, instructitmal media, and curriculum. One option was 
taught in a conventional classroom while the second was taught in a 
laboratory. The first option relied on a combination of class discus- 
sions, some lectures, and individual student c<jnferenct*s, while the 
second relied almost exclusively on tutorials, with <ine teacher-tutor 
for every six students. The first option employed traditional, printed 
text materials, while the second used a annbinat ion of a progranmied- 
learning text, a traditional text, and computer-assisted instructicm. 

The holistic option. Perhaps the most important diflerences be- 
tween the two options lay in the instructional approaches to curricu- 
lum and in the theoretical assumptions which undergirded the hvo 
curricula. The first option employed what we call a hotintic approach, 
it emphasised rhetorical and compositional principles in the c*ontext 
of whole pieces of disa)urse rather than in isolation. Based in part 
on the theoretical work of Kinneavy/^ the option attended to the 
development of writing skills in the context of three pur|K)ses of 
written discourse — the expressive purpose, the persuasive pur- 
pose, and the referential purpose — as realized through Kinneavy s 
four modes of discourse — description, narration, classification, and 
evaluation.^ The holistic approach was thus a top-down or whole- 
to-part approach to teaching composition. Students enrolled in the 
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holistic option wrote the e(iiiivalent ol'six essays (,5(K)--f)5() words 
each), not conn ting the library paper, the jonrnal, or two pretest and 
two iMisttest essays rei|nired by the evahiation design (see l)eIow). 

The fneristic option. The secH)nd optitm employed what we call a 
particle or meristic approach. Rather than working Ironi the whoh* 
dismirse to the individnal parts as in the /lo/iv/ic* option, the meris- 
tic option approached the development of rhetorical and composi- 
tional skills in a deliberately synthetic way. working from individual 
discourse parts— in particular, seiitenct s— to paragraphs and, fi- 
nally, to multiple-paragraph essays. The rhetorical theory under- 
lying this secmid option with its botttmi-up approac h was Christen- 
sens generative rhetoric as adapted by Michael (;rady^* to the whole 
essay, and taught with further modificatitnis through Wittig's Steps 
to Structure/* the major textbook in the course. The second option 
treated the sentence as the microbism of the paraurapli and the 
paragraph the inicrot<jsm of the essay. Accordingly, the assumptitni 
und(*rlyiug this option was that skills developed in writing sen- 
tences could be synthesized into the larger, more complex skills re- 
quired, first, ill writing paragraphs and. second, in writing essays.^' 
The students enndled in the se«)nd option did not write nearly so 
many essays as did the students in the holistic option. In fact, they 
wrote only two besides the four ivssays collected as part of the eval- 
uation. The students in the meristic opticni did, however, produce, 
thnnigh the large iiuniber of sentence and paragraph exercises, a 
number of words comparable to that produced by students in the 
holistic option. 

If |)ossible diBereiices in student performance were ;to be ex- 
plained, the cnrrienlar and instructional differences between the 
two options had to be accoinmodat(*d in the research design. A de- 
sign which factored out all of these differing cuinpoiients would have 
called for the examinatitm of student perfbnnaiiee in 42 different 
ctnirses, only two of which would have been of primary interest. In- 
stead, courses were constructed which w(mk\ allow five principal 
comparisons across the t\vo options.*"^' 

Five principal comparisons of the two options. The first principal 
comparison paired the two course's as tliey were normally taught, a 
ccmiparison similar to the comparis(m used in the Miami study. The 
first principal ccnnparison thus tested for overall course effect. The 
seccmd principal comparison tested for the effect of instructional 
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approach whni tin* method ot instruction was convnitional class* 
rmmi instruction for both options. In this comparison, both the 
holistic and the mcristic options were tun^ht in a coii\ entional class- 
room setting. T\\v mcristic vtmrsv examined in this comparison, un- 
like the course as it was normally taught, did not employ computer- 
assisted instruction. The third umipurison tested for the effect of 
the rhetoric text on the holistic option with hall the students receiv- 
ing rhetorical instruction thiouKh The Writiufi Cotnmitment only 
and half receiving it throuKh the c*<mrse syllabus only T\\v ihnrth 
comparison tested for the effect of instmctional method Halt tlu* 
students received instruction via the holistic approach in a conven- 
tional classroom setting and half received it in a tutorial setting. Tlu* 
fifth principal comparison tested tiir the ejfect of iustvuctioiuil ap- 
proach when instructional method was always tutorial. Th(* five 
principal comparisons are sumniari/ed in table 1. 

Selection of students and teachers. Twenty beginning freshman 
composition classes (approximatHy 5(K) students) were selected for 
the evaluation study. The mean K(T scores for the students in these 
20 classes did not dit}er significantly from the mean ECT scores for 
the students enrolled in the 16()-phis classes of the same course of- 
fered that semester. Since we wanted to he able t<i draw conclusions 
that would apply to all classes and students in the first cinirse in 
freshman writing, this test of coinparal)ility was mandatory. For 
each of the five principal comparisons four classes were used, two 
for each of the two sides of each comparison. By using two classes for 
either side of each comparison, we were able to ensure that no sub- 
group diHered measurably from any other subgroup in the study. 
And when they were cotnpared* the mean VXTV score for an\' two- 
class subgroup used in the five principal comparisons did not differ 
significantly from the mean VXTl score for all 2U sections in the study. 
Although the four classes for any giv(*n comparison were taught at 
alnnit the same time during the day. time of day was not rigorously 
cimtroiled across comparisons. For each principal couiparisoiu two 
teachers were selected, with each teacher assigned one class on ei- 
ther side of the comparison. This priK'edure allowed us to control 
the teacher variable. By having the two teachers in each of the five 
principal comparisons teach one class on each side of th(* compari- 
son, we believed that if greater effectiveness were ol)served at the 
end of the term for one side of the comparison, that greater eflec- 
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Tuhlf 1 

Summary of the Five Principal Comparisons- in tlw Texas Shidy 



C(ini|>iiri* 




leai'liiT 


\ nt 






Kiln 










Kflwt 


1(a) 


1 


I 


9 


h(ihshr/c*on\r lit iniiul 


(OviTall 




9 






holbhcVctiiiveiiUimal 


1 nc t l III 


Kb) 


3 


1 


9 


iiimsUc/luh'hitoriul 


»nirsr) 




4 


•T 


9 


iiitTistic/lalvtutDrial 




2(al 


5 


3 


9 


holishc/c*onvc*!iUiiiiiil 


mvLi lit 




6 


4 


9 


hii|jstjc7c*oiivi>iiUiiiiul 


C'Urriciiliiiiil 


2(b) 


7 


3 


9 


im*rishc/c*unvi>ntiiiiiul 






S 


4 


9 


iiicTishc'/cxinvi*iUi(inal 




■Ka) 


9 


5 


9 


hi)ijs|jc'/n)nvcM)hiinul 


IKffvvi lit' 




10 


6 


9 


hnlisljc'/cHinvenhiitial 


syllabus in 


'Mh) 


11 


5 


9 


hiilistie (svllaliiis)/ 


linlistk 










ninvi'Dtiuiial 


i iirriiiiiiiin/ 




12 


6 


9 


hiilistic (svllaliiis)/ 












UMivcnUoiial 




4(a) 


13 


7 


9 


h(ilistk'/iH)nvi*nhiinal 


mvvi ni 




14 




9 


noljstjt'/t*(iiivc*ntionul 


iiistnicUniial 


4(b) 


15 


m 

1 


9 


hnlisitk'/tiitonal 


iiieUicMi) 




16 


H 


9 


hiilistk'/tiitnrial 




5(a) 


17 


9 


9 


hnlistii'/tiitorial 


(Kflia t of 




18 


10 


9 


hnlistk/tiitorial 


('urric'iihim) 


5(b) 


IM 


9 


9 


iiicrislii'/tiitoriai 






20 
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9 
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liveness could not be attributed to teacher ditterences across the 
cx)mi}arison. 

Test procedures. The 500 students enrolled in the 20 classes were 
administered several measures pre and |)ost to determine chanfjes 
across the semester. DurinR the first three or four class perioils— 
depending on whether the class was taught on a Tuesday-Thursday 
or a Monday-Wednesday-Friday schedule—each student took tht? 
Miller-Daly Writing Apprehension Test (WAT)/" the par-graph 
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c<MtipreluMi!iii)ii sfctii>ii i)i ihv McGraw-Hill Hmliiifl Tcstr^ t\\v 
Mcllraw4lill WrUinfi Tvstr^* ami hvo writins assimmuMits— our on 
a topic which drew on personal cxiu'riciur aiul cncouraKcil narra- 
tive and descriptive details and one on an arniiinentative topic. Two 
|K«rsonal experience topics, tlie ones used in tlie Miami stnd>\ were 
used for the first oi'tlie two essays. One class on each side of each 
comparison wrote on one topic for tlie pretest while the other one on 
each side wTote on the other topic. For the i^osttest tlie topics were 
reversed. Two arKimieiitative topics were used for tlie second essay, 
one on required literacy tests ibr hiyh schiM)l seniors and one on re- 
quired hifih schiM)l a)iiiposition courses. For the pretest one class on 
each side of each principal comparison wrote on one argumentative 
topic and ime on each side wrote on the other. For the posttest the 
topics were reversed."' Near the end of the term, students were 
Kiven an opi)ortunity to complete a course-instructor evahiation 
form widely used in compositiim classes at the University of Texas. 

Analyses of data. From each of the 20 classes in the study, nine 
students who completed the course were randomly selected. The 
data collected from these 180 students were then submitted to de- 
tailed analyses. We analyzed each essay (720 in all) for length in 
words and for certain syntactic features. The four essays from each 
of the 180 students were cmled to ensure student anonymity and to 
ensure that raters could not distiimuish pretest essays Irom posttest 
essays. We then evaluated the essays liolistically. with the 360 per- 
sonal experience essays rated on one day iind the 360 arKimientative 
essays rated one week later. Kacli essay was Kiven a score raiiKiuK 
from 1 to 4 hy two raters. When the two scores ditten^d by more 
than one point, the essay was submitted to a third reading to resolve 
the difference. 

Fire detivatiie comparisons. In addition to the five principal 
comparisons, we created five derivative comparisons. These deriva- 
tive comparisons drew on the data collected from the 20 classes used 
in the five principal comparisons. In these five derivative c^)mpari- 
sons, c^mtrols over certain instructional variables were sacrificed in 
order to achieve larger "N's." The derivative comparisons were com- 
pleted Ibr the purj)ose of confirming the results of the five principal 
comparisons. The first derivative comparison pitted all holistic 
classes taui^ht in a conventional classroom setting against all meristic 
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classes taimlit in a ({mvnitiniial classruinn srttiim, 'Mir src-mul dv- 
rivativf riMiiparisnii paired all holistir classes laimlit in a nniven- 
tiinial classriMmi srttiiiu with all meristir ilasse's^tmiulit tiitnrially. 
The third minpured all classes tmmht in a cnnventinnal elassriiimi 
settinn with all elasses tmmht tutnrially. Hie fiiurth derivative nmi- 
parisnn cimtrasteil all holistic elasses with all mnistic elasses. The 
filth paired all holistir elasses tanuht in a ciniveiitiiM;al elassriMnn 
settiim with all holistic elasses tmmbt tutnrially. These five deriva- 
tive cninparisnns are sniiiinari/ed in table 2 

The analyses nl the ilata were peHiiriiied in two staues: first, fiir 
the five prineipal and for the five derivative cninparisoiis. and see- 
tnid. (or all 180 stndents eonilnned. 

Frituipal and Dvrivotin' Comparisons, In spite of the carehilly 
eoneeived research desinn. the prineipal and derivative cniiiparisoiis 
in the Texas study yielded ver> lew strikiim results. In Prineipal 
(comparison 1. whieh cniiipared the two courses as they were nor- 
nially tauuht. the analyses indicated that the hnlistie-eoiiventiniial 
eourse produeed ureater vj^xius in writiuu ijuality on the personal ex- \ 
perieiiee essays than did the iiieristie-lahoratory-tutorial eourse. Al- 
though students mi each side oCeaeh eoniparison improved in over- 
all writiim quality, this eoniparison was the only ime that evidenced 
a diHerenee owiiifi to eourse etteet. Siuiiifieant improveineiit on 
both sides of each eoniparison made the courses in the reiiiainiim 
einnparisinis appear equally etleetive in proinotiiiu writiiin (jnality, 
reuardless of ty|M» of writiiin. Prineipal Comparisons 1 and 2 indi- 
cated that the ineristie eurrieuluiii produced loimer armiiiieiitative 
essays, snwiestiimthat the modified Christensen approach reflected 
in Steps to Stntctutr helped developed jiiveiitioii skills. These es- 
say lenuth fiiidiiius were confirmed by Derivative Comparisons 1 
and 2. In addition, the fimrth Derivative Comparison, which c(nn- 
pared all holistic classes with all meristic classes, indicated sipiifi^ 
cant fiaiiis flir the meristic classes on the essay lennth of both the 
armimentati\e and the personal experience essays. The five priiici* 
pal and derivative comparisons also produced scattered syntactic 
differences that did not consistently favor eitlier currieuluiii or any 
instructional nietli<Kl. No simiifieant differences were observed for 
any of {\\v standardi/ed-test measures nor for the Writiiin Appre- 
hension Test. 
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Table 2 

Summary of the Five Derivative Comparisons in the Texas Study 
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Effect 
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All students. Analyses of the pooled data iur all 180 students showed 
a significant pretest-to-post* st change (p < .037) for only one syn- 
tactic variable, the perceuii^^e of words in final nonrestrictive modi- 
fiers in the argumentative essays. The length of the argumentative 
essays, a possible indication-of greater invention skills, increased 
significantly (p < 001). Holistic scores for both the personal expert- 
ehce essays and the argumentative essays also increased signifi- 
cantly (p < .001) over the semesttr, thus suggesting that as a group 
the 180 students were actually writing better after one semester of 
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instructioni As a group, the 180 students also sinnificantly increased 
three of their four scores on the McGraw-Hill Writirifi Test: lan- 
Ruage mechanics (p < .(X)l), sentence patterns (p < .(K)2), and total 
score (p < .001). The gains on the reading C(»inprehension part of 
the McGraw-Hill Readinn Test were less dramatic but nonetheless 
significant (p < .021). The changes in reading comprehension scores 
were further explored by examining subsets of the data collected 
from the 20 classes. Using t-tests, we tbund that reading ct)inpre- 
hension did not increase significantly for the »u'ri.vfic-coiiventional 
students (N = 18), the i/wm/ic'-tutorial students (N = 36), the stu- 
dents (N = 72) enrolled in all classes taught tutorialU', 'n the stu- 
dents enrolled in all the meristic classes (N = .54). However, signifi- 
cant changes were recorded by the students (N = 90) enrolled in all 
/io/<s/ic-conventional classes (p < .004), the students (N = 126) en- 
rolled in all holistic classes (p < .017), and Ibr students (N = 108) 
enrolled in classes taught using conventional methods and in a con- 
ventional setting (p < .019). No significant changes were observed 
for scores on the Miller-Daly Writing Apprehension Test. 

Additional anahjsen. In addition to looking at scores tor essays 
and for multiple-choice tests of writing-related skills, we also exam- 
ined student responses to an instructor-course evaluation insfu- 
ment, interviews with some students, and the failure and attrition 
rates for the 20 classes. 1 

The course-instructor evaluations were conducted using a 20-item 
instrument designed by a faculty-student committee at the Univer- 
sity of Texas and administered through the Measurement and Eval- 
uation Center. Although the results of this course-instructor evalua- 
•tion are somewhat difficult to interpret foi- classes in which students 
saw several tutor-teachers during the semester, we could find no 
systematic differences in the evaluations of the various classes. One 
of the investigators also interviewed several of the students in the 
four classes used in Principal Comparison 1, which cdit pared the 
two courses as they had been taught for several yeai s. i lie only im- 
portant difference l)etween the two type* of classes was heard in the 
comments the meristic students made aliout the problems they had 
adjusting to a course taught tutorially in a laboratory setting. Sev- 
eral of the meristic students interviewed indicated that they experi- 
enced considerable difficulty and frustration initially, but most re- 
marked that after they liad made the adjustment, they found the 
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instructional nu'thod and setting to hv a satisfactory alternative to 
regular class meetings and ti. ^iroup instruction by a single tt^acher 

Besides the course-teacher evaluations, we examined the iailure 
and attrition rates ol the 20 classes. We believed that the attrition 
rate would indicate whether the courses seemed to be meeting the 
needs and expectations of students enrolled in them. This examina- 
tion proved to be very revealing. It showed tluit in classes taught in 
conventional classroom settings by teachers using conventional 
methods oi'instruction. the combined failure and attrition rate was 
under 10 percent or about tNVo students ibr each of the 12 classes. 
The combined rate for failure and attrition jumptnl to alxnit 25 per- 
cent for the eight classes taught tutorially. For the f«)ur meristk- 
tutorial classes, the combined rate was slightly over 34 percent. 
Since the rate for meristic classes taught in a conventional classroom 
setting with conventional methods of instiuction was well under 10 
percent, we inferred that the tutorial method was the cause of the 
higher attrition rate, not the merislic approach. 

Text-internal measures and jutlfinwnts of quahtij. In the five prin- 
cipal comparisons, we were wncerned to discover, first, whether 
students' writing improved as a result of the writing instruction they 
received. Seccmd, we were interested in determining whether cer- 
tain text-internal change * might account for any increase in overall 
writing quality. We found no evidence that syntactic variables are 
predictors of the holistic sct)res of the 720 essays examined in the 
Texas study. In fact, in a series of multiple regression analyses, we 
found that in the personal experience essays, the best predictor of 
({uality .among the syntactic variables was the mean percentage of 
words In final non restrictive modifiers. And that variable predicted 
only 3.03 percent, leaving nearly 97 percent unexplained. On the 
argumentative essays, that variable predicted even less of the vari- 
ance, 0.73 percent. And these were the best predictors among the 
syntactic variables we studied. Neither did essay length turn out to 
be a very good predictor, explaining only 4.94 percent of the vari- 
ance in the holistic scores for the personal experience essays and 
only 6.52 percent in the scores for argumentative essays. Thus our 
cadre of text-internal measures of change or growth proved not to 
be terribly useful. 

Wrtftn^j apprehension. Using the data collected for the five princi- 
pal c*omparisons, we also examined the relationship between writing 
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apprehension and certain measures oi writing ahiht> and writing- 
related skills. ^'The results of these analyses suKgest that writins ap- 
prehension is related to general verbal abilities and to writing per- 
formance. They also suggest that writing apprehension is linked to 
syntactic abilities. Our investigation showed that the low apprehen- 
sive students outperformed the high apprehensives on all essay 
measures at or beyond the .(M-level of confidence, although the per- 
formative differences were less on the argumentative essays than on 
the essays based on personal experience. This latter finding sug- 
gested that perhaps the hij^h apprehensive writers in the study per- 
formed better when the %Vriting topic elicited a text not associated 
with the personal ex|)erience of the writer. Our analyses of the 
syntactic iwtterns of the high and low writing apprehensives in the 
study pointed to a tendency among high apprehensives to elaborate 
their statements U'ss fully and to use the same syntactic construc- 
tions more fre(iu;:Mitly than low apprehensives. Low apprehensives 
were found to pcHbrm better on The EiiRlish Composition Test 
(ECT) and on The Test of Standard Written Enfilish (I SWE), both 
measures which were administered to all students in the study be- 
fore the beginning of the term. 

The results of our analyses of the data collected for the evaluation 
suggested a number of conchisions. First, over the course of one 
semester, the 180 students as a group improved significantly on at 
least two types of writing. Second, as a group the students signifi- 
cantly improved their scores on two (not counting total scores on the 
SkGraw-Hill Writinfi Test) objective measures of writing skills- 
language mechanics {grammar, punctuatitm. and spelling) and sen- 
tence patterns— and on one measure of reading ability— reading 
comprehension. Third, none of the classes affected either negatively 
or pcisitively the amount of anxiety or apprefiension the students ex- 
perienced as writers. Fourth, even though Principal Comparison 1 
suggests that students in holistic classes taught in a conventional 
setting wrote better personal experience essays at the end of the se- 
mester than their counterparts in the nuristic course taught tu- 
torially in a laboratory, none of the subseciuent comparisons suggest 
that this difference can be attributable to either the greater efficacy 
of the holistic curriculum or the failure of the meristic. Subse(iuent 
comparisons— Principal Comparison 2, Principal Comparison 4, 
Principal Comparison 5, and the five derivative comparisons— 
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suggest that the difference may be attributed not to the curriculum 
but to the use of computer-assisted instruction. Fifth, since no defi- 
nite causal relationship could be established between the holistic 
scores (m either set of essays and the various syntactic indices or 
essay length, any changes along these dimensions, while often sig- 
nificant statistically, are probably meaningless. Sixth, the tutorial 
method of instruction appeared to be less effective in retaining stu- 
dents in freshman composition than did conventional methtKls, The 
tutorial method used in a laboratory setting was even less effective 
in this regard. Seventh, the use of computer assisted instniction ap- 
peared to affect adversely the peribrmance of students enrolled in 
the memfic-lal>oratory/tutorial classes (Principal Comparison I). 

These seven conclusions allowed us to answer the two ({uestions 
posed in most evaluations of writing courses and programs: Does 
the course or program seem to affect positively the development of 
writing abilities? and Is one approach to the teaching of composition 
more effective than another? The first (juestion we should have to 
answer in the affirmative since the writing performances of «ill stu- 
dent groups examined appear to have improved >iguificanlly over 
the course of the semester and since the scores on most objective 
measures of writing-related skills also improved over the same time 
period. Because the students used in the study appeared not to dif- 
fer from the freshman class at large and because there is no reason to 
believe that the teaching in the courses examined differed substan- 
tially in either quality or kind from teaching in other courses in the 
same program, our answer would extend beyond individual classes 
to the program itself.\Our answer to the first (juestions. of course, 
entails some crucial assumptions about the validity of test instru- 
ments, both essay and (i|)jective. 

Even allowing ours^'lvifs the luxury of the same assumpticms, we 
can answer the second qik'stion only with less certainty. Both the 
holistic and the mertsffc approaches appear to enhance the writing 
abilities of their students, as judged by holistic evaluations of pretest 
and posttest essays on two very different writing assignments. The 
holistic approach appeared to do a slightly better job than the mens- 
tic in enhancing reading comprehension. Tutorial instruction, with 
which the nicrivfic approach was asscK iated in the program, appeared 
less effective than classroom instruction in retaining students. 



45 



Tht' Quan'.itativc Modi'l a3 



II. 4. 2. Critiqiif of the Toxas Sttidy 

AlthoiiKh Wf hvWvw the desiun of the Texas study to bt- sitpfrior 
to the other designs we have examined, the Texas study is not with- 
out flaw or hinitation. One of the weaknesses hes with the si/e of 
the N s used in the principal comparisons. With an N of 18 on either 
side of each comparison, we can be confident that the siKnificant 
eflfects observed are indeed significant; howexer. we have no as- 
surance against Type 2 errors. " errors which (Kcur when the si/e of 
the sample is too small to be certain that apparentK nonsiunificant 
difterences are in fact nonsimnifieant. The problem of Keneralizinu 
from the Texius study is compounded by the hifih attrition rate in the 
tutorial sections, particularly in those classes that taught the meris- 
tic ciirricuUiin. While the classes were judued e(|uivalent at the 
besinninKof the term, it is aitofiether possible that the weaker stu- 
dents and those students who would have negatively assessed tu- 
torial instruction had dropped by the time of posttest. We see the 
loss of students over the course of a study as a major problem in a 
pretest-post test design. For example, the Northern Iowa study 
finished with 12 percent of the students originally in the stud\-. 

A second weakness lies in the use of syntactic measures (e.g., 
mean clause length, mean t-unit length, mean percentage of t-units 
with final nonrestrictive modifiers, and mean percentage of words in 
final nonrestrictive modifiers). We analyzeo these variables to pro- 
vide definitive evidence on the relationship between syntactic ma- 
turity and writing quality among college writers. Because this rela- 
tionship proved so slight, we were left with no measures to explain 
the variance in holistic scores. Another weakness was the use of a 
very general course-instructor evaluation instrmnent. The instru- 
ment used simply did not distinguish among teachers or classes in 
the study Another weakness lies in the use of the McCraw-Hill 
Writma Test, a test selected because of its easy access but which is of 
(juestionable validity." In addition, there are proi)al)ly better in- 
struments available for measuring reading comprehension than the 
McCrau-HiU Readiiifi Test. Finally, we see as a weakness the amount 
of class time which was re(piired to collect all the data mandated by 
the research design. At least some of that time might have been 
more profitably spent working with students on their writing. 

Beyond the obvious limitations of such objective measures of 
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"writinn-relatcul skills/* other liinitutioiis ohhv Tfxas study arc Ifs* 
obvious but more serious. From the start we attempted to overc^oine 
deficiencies we observed in previous evaluation studies (including 
our own). We attemptf*d to examine a writin){ program from several 
perspectives, represented by the ditterent measures employed. We 
did not, however, question certain fundamental assumptions about 
how to evaluate writing courses and pronrams. After two years of 
collecting and analyzing over 28,(KK) scores on various nieasures and 
categories, we discovered that we had found out little or nothing 
about what instructional practices or what composting practices 
brought alwut the higher holistic scores at the end. We began to 
question what we saw as the dominant model for evaluating writing 
c^ourses and programs. 

II. 5. An Overview of the Dominant Quantitative Approach 

To the extent that the Northern Iowa study, the San Diego study, 
the Miami study, and the Texas study are representative (and we be- 
lieve they are) of the designs and methods used in many evaluations 
of college writing programs, we can construct an evaluation model by 
fcKiising on the features common to all four studies. Most obviously, 
all four studies measured change across time and all invo|vt?d com- 
parisons. Indeed, it is impossible to envision an evaluation of any 
kind which dotts not involve explicit or implicit comparisons. In all 
four studies, comparisons were made across at least two groups of 
students, groups which apparently difl'ered with resiml to the way 
students were taught to write. The Northern Iowa study recognized 
two groups very broadly defined; the San Diego study posited four 
groups defined on the basis of large progrannnatic difierences; the 
Miami study compared two groups within the same program; and the 
Texas study compared ten groups to one another. In all lour studies, 
an evaluator's understanding of one group always depends on a con- 
comitant understanding of the other group or groups examined. 

A second shared denominator is that all tour studies |(K)ked for 
changes in student performance across time, changes which tended 
to be attributed to the particular kind of writing instruction a given 
group had received. Changes in the pcrtormance of one group are 
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interi)rt'ted in annparison witli tho periorinancr ehaiini s in at loast 
ont* other Ki'^>iip 

These* two coniinon (Jenomiiiators point to a third tcMtnrc wliich 
ail four stnchi's sliarc. Kvahiativc judKnirnts— whotlu'r •)! the efli'c- 
tivenohs of different programs iis in tlie San Die^o study or of dil- 
ferent courses within a program as in the Miami and Texas studies— 
of effeetiveness are always relative judgments, Tliat is to say, all four 
studies rendered judgments and drew conclusions which are at U'st 
vahd for only the particular pn)Krani. Huis for the Texas study, we 
can (nily claim that this or that sroup outperformed this or that Knnip 
in the Texas study. We cannot conclude or claim that a given Texas 
group performed as well or better than, say, one of the groups in the 
Miami study Similarly, while we may he able to assert on the basis 
of our ex ahiation of the Texas freshman program that the program is 
effective, we cannot claim that it is, for example, more or less effec- 
tive than any or all of the four San Diego freshman "programs," In 
addition, since neither the San Diego, Miami, nor Texas study em- 
phiyed control groups which did not undergo writing instruction, 
none of these three studies can claim with certainty that the changes 
in student performance are solely attributable to this or that writing 
course. 

In all four studies, judgments of the effectiveness of a given course 
in a writing program or the program itself are relative ones for still 
another reason. All four studies employed some fi)rni of holistic 
evaluation of pre^^^st and posttest essays to determine the amount of 
change effected by a program. As we pointed out in our discussion 
of the San Diego study, holistic evaluations of essays are evaluations 
which are dependent on the total set of essays rated or scored at a 
particular time. Ignoring for the present the common set of essay 
topics used in both the Miami and the Texas studies, we can say that 
the writing topics differed across the four studies, lu addition, there 
were probably some differences in student ability and in range of 
student ability across the four studies. Because researdi has shown 
that written products may differ according to pur|>ose. mode, and 
topic ^ and that writers of differing abilities write differently, it is not 
possible to infer that a gain in holistic score of, say 0.60 in one study is 
a^mparabie to a similar gain in another study, even when the scores 
are based on the same nund)er of scoring categories. *" Similar prob- 



48 



36 



ExaliiatiiiK C.'olleK^* WVitinu Prourains 



♦ 



lems obtain when the saiiu* topicii an* used in cliBerent studies: even 
if liopulations are comparable to one another with respect to abih'ty 
and ranges of ability, the scores are still dependent on the prol)ably 
diflPerent criteria and pnwesses by which raters arrived at their judn- 
nients. As Freedinairs research shows, even slight variations in the 
ways holistic raters arc trained can have significant effects on the 
ratings they give/* 

Another feature conuncni U) all four of the studies we have re- 
viewed is the rebancf* on products tor evaluations of student pertbr- 
mance (whether in the form of essays the students wrote or in the 
form of answers given to objective test ({uestions) for judgments of 
program efl'ectiveness^ The depeml'iicy on products jn the quan- 
titative model involves a number oi assumptions about writing in- 
struction, about the development of writing ability, and about ways 
of measuring such developnient or al)ility. First, it assmnes that the 
etiects of writing instruction on students should be evident in the 
students* written products after only a very short time. SecH)nd, it 
assumes that controlled essays or objective tests are sensitive to 
whatever learning may have (Kxuirred In'cause of the tH)urse. Third, 
such dependency assumes that the measures used are consonant 
with the goals of the course and program. 

Kach of these assumptions may in turn be challenged. It is possi- 
ble to argue that even if the development of writing abilities is ac- 
celerated by means of instruction, growth along those dimensions 
which affect writing (piality may occur so slowly as not to be mean- 
ingful after a relatively short time. Second, it could be argued that 
an instructional program in writing may affect the development of 
certain cognitive skills — such as the ability to classify — without 
having yet measurably or significantly affected tb(* (|uality of stu- 
dents* written products. Third, it might be argued that an increase 
in the quality of written pnKlucts may depend on the prior develop- 
ment of an und(*rstanding of rhetorical sittiations and the prior de- 
velopment of an awareness of and control over certain composing 
pnK*esses. A program may cause such understanding, awareness, 
and control without there bcMiig any immediate evidence in the stu- 
dent's written texts, especially those which are written under the 
ctmtrived and artificial enviroinnent of a classroom or a testing cen- 
ter. Furthermore, composition instruction often increases the innn- 
ber of Variables students nmst control as they make decisions during 
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actual writiiiR. A www highly di'vi'loped awamu'ss without a coni- 
plemfiitarv di-vi-lopinent of control ovi\ composing proccsst-s may 
have an adverse affect on students written products. The rehance 
on a nuKlel ol evahiation which depends so Vavily on the semester- 
end prcHhicts may In" a misplaced reliance. \ 

Because ol their disproportionate attt'ntion tWmester-end phkI- 
ucts. pretest-|)osttest designs Ibr writing proRrtun evaluation are 
inadecpiate for identifying and measuring chan^i^s in com|M)sing 
lu'haviors. They are also inadecjuate for evaluating x^Jiat actually oc- 
curs in the classroom. In fact, many pretest-posttest\tudies do n«)t 
consider this variable at all. In the two studies which attempted to 
assess teacher and «nirse efi'ectiveness— the San Diego and the 
Texas studies— the assessments were largely post hot onA depen- 
dent on students" recollections of things that had ocvurrecT^several 
weeks or months earlier. None of the four studies made any alitenipt 
to observe or evaluate instruction systematically as it occurred, hut 
all of the studies drew conclusions alumt the efl'ectiveness of instruc- 
ti(m. These conclusions amounted to inferences based largely on 
products collected at the end of the instructional period. For the 
same reasons that such products may not give a reliable or valit^ in- 
dication of growth in writing and writing-related skills, they may not 
have yielded valid or reliable bases Ibr inferences about instruction. 

Another characteristic of the (piantitative model of writing pro- 
gram evaluation used to date is the assumption that the goals of the 
program are appropriate Ibr the population served. In none of the 
four studies we reviewed were any rigonms attempts made to assess 
goals. Indeed, in none of the studies \wre specific goals even identi- 
fied. The lour studies simply assumed (1) that specific goals existed, 
(2) that they were appropriate for the student population served. (3) 
that these goals were being realized through the instructional meth- 
ods empk)yed. and (4) that they reflected aderjuately the larger con- 
texts ill which the programs and courses existed. 

Furthermore, all Ibur studies also ignored what might be called 
the logistical aspects of the programs and courses. None of the stud- 
ies, for example, asked whether the structtne of a program was a 
g<M)d one. whether it served adcijuateiy the needs of the students or 
the teachers. None of the studies tried to determine whether pro- 
grannnatic policies were being implemented in individual class- 
rooms. None addressed the (juestion of general teacher prepared- 
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ness for teachiiin within a nivi'ii pronrain or whotlirr thr teachers 
adequately prepared (or ehisses. Nciiie exaiiniied teaelier-studeiit 
ratios in an attempt to see whether the program or course was at- 
tempting t(M) nnieh or too htth' with the teaehinu resources avaihdiUv 
This overview ol the lour studies indicates tlie general nature ol* 
the quantitative model used in writinn proKrain evahiaticms to date. 
This mmlel has at k'ast the iiillowinn characteristics: (1) it is 4iriented 
more toward prcnhicts than it is toward prcK'esses. whether oi com- 
l>osinn or of instruction; (2) it usually attempts to measure changes 
in prmlucts over relatively short periods of time; (3) it provides fiir 
post hoc suminativc evaluations, evaluations which camiot channe 
courses or programs while they are operation; (4) it prcKluces find- 
ings which are for the most part IcKal in their applicability; (5) it is 
predicated on several untested assumptions ahout the development 
and measurement of writinn ability; (6) it does not examine the 
appropriateness of goals; and (7) it ignores program structure and 
administration. 

Evahiation studies, indudinn our own. which were based on the 
quantitative model have yielded few major jnsinhts concerninn the 
teachinn of writinn or the operation oiwritinn programs. Indeed, 
the findings ol most evaluations of writing programs and courses 
hardly justify the massive efliirts r(*(|uired to conduct the research. 
The implication for would-be evaluators of either writing proHrams 
or writing courses is clear enough; no matter how carehdly con- 
ceived and c|onstructed the design or bow sophisticated the methods 
of analysis, evaluations must be based on more than pretest and post- 
test writing samples. Evaluations ol writing programs and courses, if 
they are to result in valid and reliable judgments, must employ a 
variety of methods and prcKcdures. Neither the expert-opinion ap- 
proach to evaluation nor the pretest-posttest (piantitative approach 
will alone suffice. Most importantly, evaluations of both writing pro- 
grams and ccnirses must prcKced from a theoretical framework that 
can accommodate the complex workings of a writing program. In 
the following chapter we outline such a framework. 
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A Framework for Evaluating 
College Writing Programs 



AS TIIK FOUR STUDIES WE REVIEWED SUGGEST, THE COMPLEXITY 

of coIleHe writinH programs has pt»rbaps done more than anythiiiK to 
impede the development of adequate evahiation materials and pro* 
endures. If the quantitative model and the expert-opinion approach 
are inadecpiate, they are so, in part, because they fail to ac*c*ommo* 
dote several important c*omix)nen^s of ct)lleRe writing programs and 
the Ci>mpiex relationships among these components. Wh a> is needed 
is a framework for evaluating college writing programs tiiat can en- 
cx>mpass their dynamic nature, a framework which will help over- 
come many of the weaknesses seen in the quantitative model of and 
the expert-opinion approach to writing program evaluation. 

A framework which purports to address the evaluation of c^illege 
writing programs must he able to do at least two things: it must spec- 
ify the necessary components of writing program evaluation, and it 
must reflect the interactions among those c*omiK)nents. These com- 
ponents and interactions determine the kinds of (juestions that eval- 
uations can hv designed to answer. In the present chapter, we iden- 
tify and describe the niajor ct)mponents of writing programs with 
which the evaluator must l)e cx)ncermd, as well as the interactions 
among them. 
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IIL 1. All Overview of Five Components of Writing Profiram 
Evaluation 

We contend that there are five yeneral (H)inp()nents ot writing 
program evaluation. Within these five components are suhsnnied 
other coiniH)nents. several of which we discuss in the panes which 
follow. For the present, we will name and define hrieflx these five 
conitxments. In suhsequent sections ot the present chapter we dis- 
cuss the five coniiK)nents more fully. 

1. Cultural and wcial context, ('ultural and stK'ial context refers 
to the environment in which a writing program exists. It includes all 
influence's from outside tht^ institution which aftect either the day- 
to-day operation of the program or the nature of the program, (jd- 
tural and soc*ial ctmtext niiulit also be defined as that component of 
writinK programs over which no one directly associated with the 
program has control. 

2. Institutional context. This ctmiponent refers to such matters as 
institutional iK)licies and features which can aifcct diflerent aspects 
of writing programs and the a)urses included in them. 

3. Program structure and administration. This component refers 
to two important aspects of writing programs. First, it refers to the 
way writiniy; courses are or^ani/ed into a program. Second, it refers 
to ail administrative aspects of the proKrani not directly a part of an 
administrative structure beyond the writing program itself Among 
the aspects of a writing program associated with administering that 
program are the tollowing: teaciier training and faculty dev(*lop- 
ment programs, common syllabi, provisions for and methods of 
evaluating faculty |H'rtorinance. and the logistics of delivering writ- 
ing instruction and curriculum. 

4. Content or curriculum, (Content or curriculum is that which is 
taught in order tor the program to accomplish its goals or objectives. 

5. Instruction, This a)mpoiient refers to the methods or mean' 
used to teach the t*onleiit or the curriculum of the program, in short, 
what teachers do to help students realize the goals of the program. 

Each of these five components may interact with one or more of 
the other components. From the interactions among these five com- 
ponents, we can examine the effects of writing programs. Several 
points need to be made iniiially about effects. First, eflects in writ- 
ing programs may be either intended or unintended, but they al- 
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ways rfsiilt from interactions ainoii^j the Hvr coni|>onfnts. The in- 
tended effects ol a writinK proKrani. like its curriciilmn, are likely to 
Ih' reflected in the Koals and objectives of the program. In contrast, 
unintended ^fleets are never stated as Koals or obieetives. They are 
|H*rhaps bes^ described as unexpected or unanticipated, and they 
may be either positive or negative in nature. Unintended effects are 
more frequently identified by accident than by design. Second, 
whether inti'nded or unintended, ellects can also differ with respect 
to when they otrur. Some important effi-cts are observable only 
during the operation of the program; others may be "outcomes** evi- 
dent only at thi- end of a term; still ethers may be hMiK-ran^e effects, 
such as subsecpient academic, on-the-job, or social performance. 
This sect)nd point leads to a third: the effects of a writing proKrani 
can be seen in its students, l>oth during and after their formal in- 
struction in writing; in its teachers; in its institutional ctmtext; and 
in its cultural and social context. Finally effects can be seen in sev- 
eral kinds of data—amonK them written products, composing pro- 
cesses, and attitudes. 

Each of the Kve coinijonents. together with a proKrau) s intended 
and unintended effects, Kem>rates tpiestions which may be of inter- 
est to the evaluator of a college writing program. For example, the 
evabiator may iiei-d to determine whether the program is meeting 
the needs of the cultural and social t^jutext. The evaluator may also 
he called upon to determine whether any or all aspects of the pro- 
Krani are, Ibr example, tH)mpatible with the nussion of the institu- 
tion. Other evaluative (piestitms mi^jht fotiis on the appropriateness 
of a program s curriculum o* the instructicmal methods used to teach 
it. Answers to these ({uestions — as we suK^ested in our revit^w of 
the Northern lowa» San DieKo, Miami, and Texas studies— are most 
often discussed in terms of the effects of writing programs. By and 
lar^e, the effects which evaluations of college writinj^ programs have 
focused on are intended effects, usually measured through examina- 
tions of students* written products. Indeed, none of the four studies 
we reviewed provided for an evahiation of unintended effects. 

The evaluator usually assumes that the first four a)mponents of 
writiuK program evaluation— cM/fwra/ and social context, institu- 
tional context, program structure and administration, and curricu- 
lum — will remain relatively stable. Of course, the histories of many 
colleges and universities durinn the Viet Nam era illustrate how 
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contextual influences can channe from day fo d>?y» as do school dos- 
ings owing to natural gas shortages. In addition, unplanned chaiiges 
in program personnel, either administrative or instructional, may 
result in administrative or curricular changes. Imtrtiction, on the 
other hand, is usually characterized l)y change since instruction is a 
dynamic process. Teachers frequently change direction or tempo as 
they adjust to the needs of particular 'indents. Few teachers can 
plan the instructional strategies of an entire term without having to 
make adjustments before the end of the term. These changes con- 
tribute to the diiliculty of assessing instruction, whether within in- 
dividual courses or across courses within programs. Accordingly, 
evaluators must be concerned with ways of describing and evaluat- 
ing instruction as it (KX'urs. 

Insofar as a program seeks to alter the knowledge and performance 
of its students, the program s effects may themselves be process- 
oriented variables. Indeed, when later in this' chapter we discuss 
the intended effects of writing programs, we opt for the word per- 
fonnance instead of, say, the term product or the term achievement 
to describe eflects with reference to students in programs. Perhaps 
from Tylers early work ou curriculum and evaluation,^ the term 
achievement has c-ome to be associated in evaluation studies with a 
model which is primarily concerned with static products,^ It is tiiis 
"static products" or "outcomes" model which provides the underly- 
ing structure for the various National Assessments of Educational 
Progress and for the competency testing movement.* The "static 
products" model is also reflected in the quantitative method we 
identified in our reviews in chapter 2 of the four evaluatitm studies. 

But in a writing program, products are simply the results of perfor- 
mances or prcK-esses. In writing tt)urses and programs, we believe as 
much, if not more, attention ought to l)e paid to how prodi^v'ts come 
to be as to products themselves — that changes in prwesses of com- 
posing should be considered as legitimate an efl'ect of writing pro- 
grams as changes in the products of composing. It is not so much 
that products are unimportant but that products in writing courses 
are most meaningful when they are viewed as aspects of proces; • 
(; »nted performances. The intended efl(i*cts of a writing program 
may also be reflected in the attitudes students bold — attitudes about 
writing, about language in general, and about writing courses — as 
well as in teachers* attitudes toward themselves and what they do. 
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To better understand how intended and unintended effects come 
about, we will take a closer look at the five major components of 
wnting programs. 



Iir I. 1. The Cultural and Social Context of Writing Programs 

The cultural and social context is an extremelv im|>()rtant compo- 
nent oi writing programs, for it imposes certain restrictions or con- 
stramts on what the writing program can do. Cultural and social 
context aftects the nature and the number of educational goals and 
objectives; it partly determines the curriculum taught; it influences 
how teachers teach and how effective they can be; and it contributes 
to student periormance. The influence of the cultural and social 
c-ontext can be as substantial as dictating the specific goals of the 
program and as insubstantial as influencing the places where stu- 
dents can purchase text materials, as well the availability of those 
materials. The whole notion of a freshman composition program as 
a service to other academic departments derives in part from the 
uifluence of cultural and social context. In fact, a recent survev of 
writmg program directors and teachers fbund that many of these 
academics thought society at large wanted them to serve s,)cietv bv 
producing students who could write •'mechanicallv correct" prose 
up<m graduation. * The "Back to Basics" movement probably was the 
direct result of the influence of social context on the teaching of 
writing in colleges and universities. 

In the widest perspective, the context includes (|uestions alH)ut 
the value and uses of literacy in a given society Much recent work 
has examined these questions, fbr the most part looking at the effects 
of hteracy in societies undergoing modernization.' But there has 
been relatively little work done on the functions of literacy in a mod- 
ern, technological society or why college students should be taught 
composition skills." In the first decades of the United States as a na- 
tion, writing was not so much essential to the working lives of Amer- 
icans as it was to their swial and political lives.^ I„ contemporary 
American society, however, writing appears to be far more useful as 
a vocational skill than as a social skill, at least fbr college graduates " 
Other influences of the cultural and social context may be more 
subtle. In state institutions, fbr example, class size is often deter- 
mined by the amount of money legislated fbr the operation of the 
institution. Another important aspect of context is the students at- 
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tending an institution. Which students enroll in the courses iti a 
writing program is often determined by policy- and decision-makers 
who are not themselves directly involved with the program itself. In 
state-supported institutions, legislators— sometimes in concert 
with administrative officials— determine which students and how 
many ol'them can be admitted to a particular cH)llege or university. 
The abilities of these students, as well as their own career goals, can 
influence the shape of a particular writing program. Postlethwaite 
cites evidence from the first National Assessment of Education Prog- 
ress^ of variations in the backgrounds of students which alf'ect the 
ways students are able to learn, conditions which ultimately in- 
fluence the way disciplines at the college level can be successfully 
taught.*" \ considerable body of research suggests that such con- 
textual variables as students* academic fields can, independent of 
ability, afl^ct the ways students approach tasks in other fields;'' and 
Frederiksen's work on "task structure" indicates that strategies em- 
ployed fur different tasks may be determined both by the nature of 
the task and the students ability,''' which may difier according to ed- 
ucational or cultural background. In addition. Havigburst argues 
that the particular roles people play at difterent ages from preschool 
through retirement aflect what they expect from education at dif- 
ferent ages.'^ Anyone who has been privileged to teach older stu- 
dents with full-time jobs and families to support can appreciate the 
c*(mtextual influence which Havighurst identifies. 

While it is not our purpose here to delineate the many ways'* by 
which the larger cultural and social cxmtexts of writing programs 
may influence the programs themselves, we do wish to illustrate the 
importance of these larger contexts. Perhaps the most obvious influ- 
ence comes in the fbrn) of admissions policies. Virtually all educa- 
tional institutions have altered their admissitms policies within the 
last decade. The classic example of such policies — which, in turn, 
reflect cultural and societal values— directly influencing writing in- 
struction is the case of the City University of New York. Once a uni- 
versity geared primarily to teaching students with above average ac- 
ademic preparaticm, CUNY, because of society-mandated changes 
in admissicms policies, became responsible for teaching large num- 
bers of underprepared students. When CUNY's adniissicms policy 
became one of open admissions, new ways of teaching students to 
write competently had to be developed. Out of this need to ofler 
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writinK instruction toVsiihstantially diflerent student population 
came the now well-known work oi'Mina ShauKhnessy and a host of 
others.*** 

It seems to us that any comprehensive evaluation of a collefje 
writinn program must explore the cultural and social contexts in 
which the program exists, for part of the effectiveness of a program 
is directly tied to the extent it is responsi\ e to the cultural and social 
context. Only through the examination of larger contextual influ- 
ences, which composition teachers and writing program directors 
often view as constraints, can one arrive lat an adecpiate understand- 
ing of how and why the program operates as it does. 

III. 1. 2. The Institutional Context of Writing Programs 

The institutional context of a writing program may often he as dif- 
ficult to pin down as its cultural and social contt xt, because a writing 
program is itself a part of its instituticmal context, constantly shaping 
otht i- features of the institutitm and constantly being shaped by 
those features. 

Sometimes the institutional context can be seen as the constraints. 
iKith positive and negative, which are impo.sed upon the writing 
program by the institution in which it exists. Many of these "con- 
straints" take the form of institutional policies; others are reflected 
in traditional practices, which may exist independently of any stated 
policies. As Rutman has argued, perhaps the best way tor the eval- 
uator to get a handle on the institutional context is to examine docu- 
ments^' which outline, among other things, the operating proce- 
dures of the institution. For instance, the evaluator may want to 
examine the institutional documents relating to the responsii)ility 
and authority of the writing program director to determine if the 
institution ac{iially allows that administrator to direct the program 
he/she is responsible for. 

In some cases an examination of policy documents may lead to 
evaluative judgments directed at different aspects of the institu- 
tional context, judgments rendered in hopes of changing that con- 
text. One of the more common complaints offered by the writing 
program directors who responded to a recent national survey was 
that the attitude of teinired professors toward the teaching of writ 
ing fostered negative attitudes among junior colleagues and helped 
to keep the writing program at a second-class status within the de- 
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partinent, usually an EiiKlish departineiit.''* Thv second most com- 
monly cited problem was the inefiectiveness oi the administration 
of the writinK pronram itself, an inefiectiveness caused. aca)rdinK to 
several directors, by tlie departmental limitations imposed on the 
positions they hold. Both of these problems sunnest that whatever 
is wronK with the writing program itself can only be corrected by 
changing the basis on which laculty are rewarded and by chanKing 
the priKedures under which writinn program directors are forced to 
operate. In these two instances, an examination of the institutional 
documents — such as faculty promotion materials — may yield more 
than mere information that the evaluator could use as background 
for an evaluation of a writing program. 

bistitutional context can influence writing programs in more di- 
rect ways as well. The current interest in cross disciplinar>' writing 
courses also results, in part, from pressures brought to bear on com- 
position programs from the outside, in this case academic depart- 
ments that l>i»lieve students may actually learn more about a sul)ject 
by writing about it.^*^ 

Aspects of instruction are also subject to the influence of the in- 
stitutional context. Although most of the research in this area has 
focused on instruction in elementary and secondary schools,*"' it is 
reasonable to assume that some of the same institutional influences 
afiect teaching in postsecondary schools as well. Among these possi- 
ble influences are the physical layout of the classroom, the attitudes 
within institutions toward the teaching of writing, the availability of 
duplication facilities and supplies, and the availability of facilities 
for conferences with students. 

Adding to the complexity is the fact that a writing program can 
also shape the institutional context. For example, in many institu- 
tions all beginning freshmen are required to take writing courses. 
To the extent that these courses aflect the way students perform in 
other courses, the writing program may be said to influence aspects 
of the institutional context beyond itself Another potential source 
of influence is the size of most writing programs. In many institu- 
tions, the composition program offers more classes and uses more 
faculty than most major academic departments. Because of its size, 
the writing program may directly influence the way available re- 
.sources are allocated within the college or university and may influ- 
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ence decisions and retoniineiidatioiis urowiiiu out oi suc h yroiips as 
faculty councils and senates. 

III. 1. 3. Program Structure and Administration 

Profiram structure and (uhninistratkm is likely to present the 
evahiator with a numher oi difficulties. Most oi these problems tan 
he attributed to thi' fact that most colleye writing programs are in 
many respects unicjue, a point made repeatedly by directors who 
responded to a remit national survey.*' Many of the directors sur- 
veyed believed that their particular programs were designed and 
are administered to accommodate the needs of particular kinds of 
students and particular kinds of faculty, thus making it difficult to 
discern important or substantive common denominators across dif- 
ferent programs. Unlike writing program curricula, which often re- 
flect dif^erences in degree rather than kind across programs, pro- 
gram stnicture and administration can be uni(jue to a particular 
institution. In some programs, Ibr example, the writing program 
director may be responsible for creating and implementing pro- 
grannnatic policies; in other programs, the director may simply be 
charged with carrying out the policies handed down by, say a de- 
partment head or a curriculum committee. In some cases, writing 
program directors can determine which teachers will t(»ach writing 
courses; in other programs, the direct(n- has no control over hiring 
practices at all. In some writing programs, teachers are allowed to 
teach whatever and however they wish; in other programs, teacluTs 
are required to follow a common syllabus which may dictate both 
curriculum and instruction. In some programs, directors are re- 
sponsible for (^valuating the faculty teaching in that program; in oth- 
ers, evaluation is conducted by administrators or faculty committees 
not associated with the piogram directly In some eases, directors 
are very amcerned about thelretationships among courses within a 
program; in other cases, such relationships are not of great concern. 

The point of all this is that program structure and adviimstration 
are very difficult to describe with reference to othei programs, pri- 
marily because this component of writing programs is most sensitive 
to the local circumstances. When generalizations across programs 
are either difficult or impossible, evaluation becomes even more 
difficult. Kvaluators must not only spend considtTable time describ- 
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inj{ this complex compoiu nt^ but they nnist thtMi determine whether 
it is efi'ective within the larger context. 

III. 1. 4. Writing (^iirricnlnni: Focus on "What" and "How" 

The curriculum of a given writing program may inv()lve a nimiher 
ol curricuhir variables, primarily because writing itself is a complex 
activity demanding a larger miniher of goals and objectives. Be- 
cause writing curricula are complex, Kyle's distinct i(m l)etween 
"learning that" and "learning how"" i.s extremely useful. C'urricular 
variables may be classified either as content in the usual sense of a 
body of knowledge to be learned or as content in the sense of pro- 
cesses whkh can be applied or used to realize specific ends. In the 
courses which make up writing programs, curricniar variables are 
fre(|uently of both kimls.^* Many writing courses have as (me of their 
goals the teaching of different bodies of kfiowledge as well as teach- 
ing the act or process of writing. One bod>' of knowledge consists of 
rules governing the use of punctuation marks, definitions of kinds 
and types of written disc*ourse. lists of transition words, the concept 
of an essay or a paragraph. Other bodies of knowledge (e.g. . literary 
genre» themes such as death and dying or humor) may provide the 
bases for what students can say analytically or inter])retively about a 
written text. If a writing course curriculum retpMres students to ana- 
lyze or interpret written texts, it delivers a content in tlie sense of 
processes, processes possibly (piite diflerent from those engaged 
during the act of writing. Such pnKesses are. of course, themselves 
cnrricular varialiles. In conducting evaluati(ms of writing courses or 
programs, it is always desirable and frequently necessary to rec^og- 
ni/e not only the cnrricular variables of the course or program l)ut 
also the way or ways these variables interact witli one anotlier. For 
example, the students* knowledge of the history of the modern short 
story, their knowledge of the principles of literary analysis, and their 
knowledge of composing pnKesses may interact in peculiar ways to 
produce better, or worse, freshman writers. 

Klements of composition curricula have received much attention 
in the literature. Indeed, virtually everything ever said about what 
to teach in writing courses falls imder the general rubric of curricu- 
lum. Subsumed imder this general rubric is anisiderable diversity, 
much of which reflects a niore-or-less traditional paradigm.^* One 
usehil way of understanding curricula is through the textbooks used, 
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and perhaps followed, in the cinirses which make up the program. 
The importance of examining texth<H)ks is siignested l)y our recent 
survey e' writing program directors. In that survey we found jiireat 
variation across types of institutions in the kinds of texthniks used 
in first- and second-semester freshman composition courses/'^ 

VVilham K W(H)ds. in a recxMit review article.*^ proposes a useful 
taxonomy for classifying coIIeRe writing textl)(M)ks, Woods initiall> 
distinguishes het%veen discipline- and student^entered texts, both 
of which make certain key assumptions about what is to he taught in 
the txjurse. Acctirding to VVikxIs. student-centered texts are gener- 
ally expressionist in nature, stressing the natural development of 
the student s ability to express his/her thoughts and feelings in writ- 
ing. Acvording to Womls. ^expressionist" texts regard all aspects of 
commmiication as extensions of the writer: language becomes the 
"'writers voice. " audience becomes the " writers inner ear/" and 
logic becomes " the writers vision " (p. 397). Discipline-centered 
texts, in ctnitrast. are "interventionist" in that they assmne the in- 
dividual needs to he shaped by cultural tradition. From this per- 
spective, discipline-centered texts are either language-based (fin us- 
ing on such matters as usage and style), rhetoric-based (focusing on 
traditional or modem c^ontvrns of rhetoric), or logic-based (focusing 
on the imi>ortance of clear thinking). Woods provides a numl)er of 
subclasses under both language-based and rlietoric-l)ased texts, but 
these subclasses need not ccmcern us here; what is important is that 
while each type of text may have certain features in nnnnion w ith 
other types of texts, the diflerences among them may very well sig- 
nal important differences in the curricuhnn of which they are a part. 
For example, if an "expressionist" text inffiuMices a writing curricu- 
lum, then one would expect students to do mori* writing from per- 
sonal experience.^' perhaps in the form of narrations, than they 
would if the controlling text were a discipline-centered text. 

To distinguish among curricular elements on the basis of what 
and how. we may choose to view the coni|M)nents of a composition 
curriculum, as Frank J. D'Angelo does, in terms of an "underlying 
structure" for tt^oiposition, a structure that "can be conceived of in 
terms of principles and forms."^' D'Angelo argues that three sets of 
discourse principles and two sets of disaiurse forms provide this un- 
derlying structure. For D'Angelo three general categories of dis- 
course principles— mechanical, linguistic, and rhetorical— underlie 
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the touching of composition. Each of these );c*iieral catenories itself 
contains several principles. Inchided nnder the rubric of "niechani- 
car* principles are handwriting, capitalization, punctuation, and 
spellinft. Within the class of linguistic principles, D'AnHelo inchides 
the principles of the word, sentence, paragraph, and essay; and three 
of the five classical divisions of rhetoric — invention, arranfiement, 
and style — constitute D'AnHelos class of rhetorical principles. These 
three categories of discourse principles rt*present, according to 
D'Angelo, '*the fundamental laws, rules, and conventions of dis- 
course" and together determine "the intrinsic nature of discourse.'* 
In the hands of a writer addressing an audience for a punw>J><'. these 
discourse principles combine to produce discourse forms and are 
themselves reflected in those forms. The forms of discourse can. ac- 
cording to D'AnHelo. be either the traditional discourse modes — 
description, narration, exposition, and persuasion — derived from 
Alexander Bain *" or the modern aims of discourse — expressive, per- 
suasive, literary, and referential — us developed by Kinneavx. 
D Angelo s "underlyinn structure" of composition can serve evalua- 
tors as a very general guide for identifying certain of the curricular 
variables in a composition program, even though Bain's and Kin- 
neavy s theories of **forms" are not as easily rett)nciled as D'Angelo 
suggests. 

Other useful and important ways of viewing the forms of discourse 
ccniiposttion curricula address appear in the work of MoHett." the 
work of Britton and his colleagues at the University of London School 
of Kducation, and the work being carried out under the auspic-es of 
the Council of Europe." Moffett distinguishes between kinds and 
(irtferv of disamrse. These distinctions are based on the relationships 
among audience, writer, and subject. Moflett's orders are "interior 
monologue," "ctmversation." "corresptmdence," "public narrative," 
and "public generalizatitm." a progression signaling increasingly 
greater distant^'s Wtween writer and audience. Moflett s kinds of 
nonfiction discourse are "drama." "narration," "exposition," and "ar- 
gumentation," a progressitm which accommodates greater distances 
i)etween writer and subject. Like Kinneavy s theory of aims and 
Mofl'ett s theory of orders. Britton s theory of functions is predicated 
on changing distances, both physical and psychological, between 
writer and audience or among writer, subject, and audience. His 
progression of discourse functions may be represented as tbilows: 
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fxpressive discourse, ptn^tic discourse, persuasive discourse, and in- 
formative discourse. Van Ek, xvorkiuK with the Council of Kurope, 
specifies six general functions for users of the English lannuaKe: "im- 
parting and seeking factual information**; "expressing and finding out 
intellectual attitudes/' "emotional attitudes," and "moral attitudes"; 
"getting things done (suasion)," and "socializing" (p. 25). It is impor- 
tant to note that these theories have several features in common with 
one another, features which suggest that the theorists are as much 
concerned with the act of discoursing as with discourse as a product. 

Evaluating a writing program curriculum on the basis of discourse 
forms and principles may presuppose the prior knowledge of sjie- 
cific goals and objectives. If the specific goals ami objectives of a 
course or program dictate that such principles and forms be taught 
either as content in the sense of what or as content in the sense of 
hou\ then curricular materials need to be examined carefully to de- 
termine how compatible they are with those goals and objectives. 

III. 1. 5. Writing Instnictitm: Se<|uences, Methods, and Media 

Instructiomd Seifuences. Not only is it possible to examine and 
evaluate writing pn)grams on the basis of the curricular components 
reflected in instructicinal materials, but it is also possible to do so on 
the basis of the wa\' curricular components are secjuenced for in- 
structional purposes. Kinneax y provides a useful system for exam- 
ining composititm instruction on the basis of the se<iuence of curricu- 
lar elements. He offers a model of instruction that consists of eight 
components arranged urouiid a circle representing instructional se- 
(pience. The.se ctmipcments are "examples,*' "analysis," "princi- 
ples," "envinmment and stimulus," "think,** "write/* "talk,** and 
"rewrite." Not all of these n)mponents arc. of course, present in 
every instructional situation. In what Kiimeavy labels the "Tradi- 
tional: Deductive Approach," which reflects the influence of classi- 
cal rhetoric, especially Aristotle and Cicero. the student begins by 
studying rhe^)rical principles, reads examples illustrating those 
principles, analyzes the examples, receives a writing stinuihis, 
thinks, writes, perhaps talks about his/her writing, and perhaps re- 
writes. In the "Traditional: Inductive Approach,** which echoes tbe 
practices of Isocrates, tbe curricular seijueiicr is somewhat dif- 
ferent: the student studies examples or models, analyzes the mod- 
els, discovers their rhetorical principles, receives a writing stimu- 
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1ms» thii)ks alxnit what lic/sho is to writf. writi's a pio(t\ perhaps 
talks alioiit the pii'cc\ and perhaps rewrites the pieee. Usiiin these 
same eight components » Kiimeavy outlines other approaches to 
teaching coinpositiun as well: "Kxample-Stiinulus" "Imitation."*'* 
"Behavioral/**** and **lA\m\ By Doing. The point of all this is that 
instructional sec|iiei)ce can. and prol\il)ly does. aHect how students 
learn to write in a composition ccnn se. Orgiuii/iitional structures, 
such as those outlined hy Kinneavy. reHect difl'erent assumptions 
al>ont the way language skills are learned and should he taught. 

Although other comiKiiients niay need to he added to Kiimeavy s^ 
we helieve that ditt'erent instnictional organi/;itions make diBerent 
assumptions al)out the content or the curricuhim ot a writing course 
or program. A g(M>d illustration o( this point is ( *liristensen*s **genera- 
tive rhetoric/' " the instructional se(pience tor one ot the courses ex- 
amined in the Texas stud>. In generative rhetoric, one of the under- 
lying assumptions is that sentences are structural microci)snis of 
paragraphs and that paragraphs are structural inicrocosuis of es- 
says.** Because of its movement Irom parts ot whole disanirses (sen- 
tences) to larger units of discourse and, finally, to whole disccnirses. 
it is ))ossible to describe the structure ot a ^'generative rhetoric * 
course as meristic. Not all writing courses are. needless to say, so 
structured. In fact, Shuy has rett»ntly argued on theoretical and 
pragmatic grounds against what we have called the meristic ap- 
proach." Some writing ctmrses are structured in just the opix)site 
way, focusing first on the whole discourses to provide a textual con- 
text for exploring disniurse particles. As in the Texas study, these 
courses might be termed holistic because of their whole-to-part ori- 
entation. Still other courses — probably most writing courses — tall 
somewhere in between the two v\t\ les of meristic and holistic. 
The claim we wish to make is that t! instructional organi/xition of 
curricular components may atlect the nature of the program itself 
and how well the students learn in a particular course within that 
program. 

Identifying the usual patterns of instructional secjuence is not dif- 
ficult. The evaluator can usually identify secjuence jjirough such 
dc»cuments as program or c^mrse syllabi, instructionii)4iaiidout^. and 
assignment sheets. However, as far as we km)W. there are no al- 
together adecpiate methods tor evaluating iastructioiial sequence. 
One reason tor the absence of adecpiate methods is the concomitant 
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ahsfncr (if roscarih on instnictional stMiuriKc in collcKr-lcvcl writ- 
iiiK ctMirsfs. For fxainplt*. we know of no convincinu evidence that a 
meristic secjuence is better than an holistic one. or vice' versa, Al- 
thoiiuh the Miami sentence-conibininu study and tlieTSorth Dakota 
study of Ueneratix'e rhetoric sufi^est tlie ureatCT efficacy of meristic 
secptences. the Texas studx' suflfiests that holistic and meristic se- 
(juences are e({ually eHective Ibr certain kinds of students. 

We believe that the success of any instructional sequence may be 
directly continuent on the counitive devc*lopment of the students 
the sequence was implemented to serve. Some of the best research 
done on teaching writing to developmental students 'Mias important 
implications Ibr both writtnu curricula and the instructional se- 
({uencinu of curricular elements in basic writinu courses. We do not 
know, however, whether the same implications would hold Ibr non- 
deveh)pmental students. The appropriateness of an instructional se- 
(juence must l)e determined, first, in relation to a writing curricu- 
lum, which beuins in the elementary Urades for individual students, 
and, ultimately, in relatitm to student peribrmance. Instructional 
secj'.ience is one area which could benefit enormously from a lew 
carefully desiuned and controlled experimental studies. 

Imtruciional Methods, If composition prourams vary with re- 
spect to contexts, curriculum, and instructional secpience. they may 
vary even more so with respect to instructional methods. Indeed. 
Joyce and Weil have identified over 80 different models of teach- 
inu- Some of these models are not. of course, ones which are wideK' 
used in con)t)osition teachinu. hut the ptnnt is that diversity in t(*acli- 
iuK practices can complicate writing prouram exaluation. This diver- 
sity can be attributed, in part, to personality diHen*nces auumu 
teachers.'* In the teaching of composition, one of the most com- 
monly used methods of instructitm is that which miuht be labeled as 
"traditi(uial classnH)m discussion"; but even within that ueneral 
method can be tbund ccmsiderable variety. For example, the discus- 
sion metluKi is lre(|uently supplemented l)y student conferences or 
writers' workshops. And within 'he area of discussion, methods may 
vary considerably. Kecentb; one of the present authors visited two 
writinu classes, each tauuht by a ditierent uraduate student enrolled 
in a practiciuu on teaehinu freshman composition. In both class(*s. 
discussion centered on assigned readinus illustratinu the principles 
of classification. In one class, the teacher led the discussion from 
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a set of questions she had prepared over tlie readings, (juestions 
fcKntsing on classification systems and nienil)ersliip in classes illus- 
trated in the readings. Students responded hy attempting to answer 
the ({uestions directly, and discussion ensued only when a particular 
res{)onse to a (piestion was deemed unsatisractory In the otlierclass^ 
the general instructional method was also discussion, but the lorm 
ol the discussion was more obviously inductive, with the teacher 
derivinf^ each sul)se(|nent cpiestion from student responses to pre- 
vious questions. While both classes may be labeled as discussion 
classcs/the actual discussion practices and pnK'Cilures diilered sub- 
stantially. The problem that writing program evaluators have to (ace 
is that if the two classes are compared on the assumpticm that both 
employed like instructional methods and one course proved more 
eHective in teaching the principles ol classiKcation, that ditierence 
in ettectiveness could be attributed to the two diiierent methods of 
conducting class discussions. The evahiator must, therefore, be sen- 
sitive to what may seem to be subtle dilferences in instructional 
meth(Kls. 

in the teaching of composition, a lar^e number of nontraditional 
methods are also frcciuently used, many of wliich liave received 
dcx^imentation in the literature. HeHectiuK both curricular and in- 
structional concerns, one useful recent summary of classroom prac- 
tices in the teaching of writing su^^ests that modern composition 
clas.«ir(M)in practices put greater emphasis on "cooperating in the 
process of writing rather than on criticizing the products of compos* 
iuK than did the practices of earlier years. This shift in emphasis 
in classroom practices — wliich also rellects corresponding shifts in 
curricula and contexts aiul makes diiierent deniands on program 
administration — may be attributable, in part, to the inHuence of 
the Dartmouth (^'onference with its concern for classnM)m interac- 
tion and activities of learning.*' At any rate, the result of this shift in 
emphasis has beiMi the developm>4)t of new instructional methods. 
Of these new methods — most of which locus on writing processes — 
two will serve to su^^est some of the problems such methods pose 
lor the evahiator of writing courses. 

One widely publicized method is the one advocated by Murray. " 
This method, as described by Murray and (Jarnicelli. involves con- 
ferencing with individual students about their writing as it is beiuK 
written. The conferences are presumably conducted so that the focus 
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ol' attention is always on the stiidiMits' writiiiy, with the tcadier scrx • 
init as a Kuiilr to help students throiiuh the writiiiy proeess. 

A second relatively new instnictional method falls under the yen- 
erai nibric of n)llaborative learning, whieli includes the notion oi 
peer tntorinu as outlined by (larrison^^ and as popularized by. for 
example. BruBee.^* Like the eonfereneiny approach to writiny in- 
struction, collaborative learning places a yreater emphasis on the 
pnKrsses involved in composing. When the ct)llaborative model is 
used, students presuniabK* have ready access to peers for responses 
to their written products and for consultation as drafts are beiny 
written. Both the conferencing approach and the collaborative ap- 
proach would seeuK by their very nature, to encouraKe student 
writers to dex elop a sense of the audience that will be reading and 
res|K)ndinK to the texts they pnKluce. 

While these two approaches, two instructional methods, to teach- 
iiiK writing are widely known and often used in college writing 
courses,'^' considerable variety is possible within both. For people 
interested in writing program ('valuation, these variations within 
general classes of instructional methods are perhaps no less impor- 
tant thaii ditterences between the classes themselves. We have seen 
peer tutoring, for example, used in a number of diHerent ways — as a 
part of a writing course primarily dependent on more traditional in- 
structional nu*thods, as a part of a laboratory course in which peer 
tutoring supplements the instruction oHered by the teacher of rec- 
ord, and as the only means of providing students with evaluations of 
the written texts they produce. So too with coulerencinK. Our con- 
cern here is, of course, the concern of the evaluator for describing 
accurately instructional methods used in writing courses. Karely. in 
evaluating instruction in any discipliiu* is it suificient to say that a 
course was taught according to this or that general model of iustruc- 
tion. Accurate descriptions of instructional methods used in college 
writing courses and programs must precede evaluations of thcMU. '**' 

Instructional Media. The components of writing curricula and 
the instructional methods used to teach writuig curricula nnist m)t 
only be kept separate from one another in the evaluation of a writ- 
ing program, but they must also be distinguished from instructional 
media. Instructional media may be thought of as delivery systems, 
and they int(*ract in the classroom with both curricular compo nents 
and instructional methods, I'ntil recently, composition speci.dists 
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needed to be concerned primarily with but two instnictioiial media, 
the printed word and the spoken word. Now. composition special- 
ists have to be mncerned with other media as well. It is liecominn 
increasin^^ly common in composition courses, ibr example, to use 
television shows and motion pictures to provide the content Ibr stu- 
dent writing or to illustrate various principles of communication.^* 
It is also becoming more common in some cpiarters to use comput- 
ers both to teach students conventional usage and to assist tnem in 
the actual writing of essays by providing for interactive ust? pro- 
grams in invention or essay structure and development.^' As in the 
use of instructional method^^. considerable variety exists in the use of 
instructional media, even ui the wax:, • particular medium is used. 
Again, it is not sufficient in writing program evaluation simply to 
identify courses which use computers, because what is important is 
the precise way computer's fit into the c»ourse itself and the pur|)oses 
for which thev are used. 



III. 2. The Effects of Writing, Profirams 

Virtually all evaluations — whether of writing programs or of any- 
thing else — have to be concerned with eflects. In evaluating writing 
programs, we must be concerned with the main ways those pro- 
grams aflect people — the students, the faculty, and even the swiety 
at large. In the four evaluation studies reviewed in chapter 2. the 
eflects of writing programs were evaluated almost exclusively in 
terms of students' written products. As we pointed out, however, 
pre(K*cii|)ation with written products in writing program evaluation 
has resulted in a narrow definition of student performance — one at 
(Kids with much of the current research on composing.'^'* In addition 
to I<x3king Ibr programmatic ett'ects within student products, evalua- 
tors must also examine changes in the attitudes of both students and 
teachers brought about by the day-do-day operation of the program. 
The preocrupation witli student products has diverted attention 
from not only the intended eflects of a writing program as expressed 
through statements of goals and objectives, but has also obscured 
the unintended eflects of writing programs. It is important for eval- 
uators to realize that whether the source of data is products, pro- 
cevsses, or attitudes, the eflects they measure may be either intended 
or unintended. 
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The intended effects of a c^ollege writiiiK program are typically 
either expressed or implied in statements of noals. Ideally, these 
statements serve as links between contexts on the one hand and 
curriculum and instruction on the other One oi the tasks oi the 
evaluator of any program — whether a writing program or a football 
program or a program for aid to dependent children — is to deter- 
mine what the program is supposed to achieve, its raison detre, 
Unlc'S.s the intended effects of the program are known, it is impossi- 
ble to determine whether they have been reali/.cd. A second eval- 
uative task, then, is to determine whether the effects intended were 
actually produced. Since some effects may be unintended, a third 
evaluative task is to identify those which occurred. 

Intended Effects, The literature on evaluation suggests two ap- 
proaches to identifying the objectives of a program. The least used 
is the goal-free model of evaluation proposed hy Scriven.^** As House 
points out» Scriven's concern with goal-free evaluation is a concern 
for the reduction of bias. Scriven has argued that if informed of the 
goals of a program in advance of the evaluation of it, the evaluator 
will likely i;verlook important "side effects," what we have labeled 



even general descriptions of the intentions (i.e., the goals or objec- 
tives] because it [sic] focuses your attention away from the 'side 
effects* and tends to make you overlook or down-weight them."^^ 
Employing Scrivens goal-free approach, an evaluator would dis- 
cover the goals, both the stated and the unstated, during the course 
of the evaluation itself 

Scriven s arguments in favor of goal-free evaluations notwithstand- 
ing, most evaluators try to identify the goals and ofjectives of the 
program during the beginning stages of the evalua^on. Although 
most writing programs appear to operate on the basis of goals and 
objectives, statements of them are typically very broadband very 
general and, thereibre, of limited value in determining the exact na- 
ture of the program or the courses within a program. Broad, general 
statements of goals have much more in common with statements of 
good intentions than they do with j:pecific, observable aspec*ts of 
writing programs. Most writing program directors and most teach- 
ers of writing, for example, see the major goal of freshman writing 
programs and courses as the improvement of stud ^ writing.''* How- 
ever important such statements may seem, the\ -ovide the evalua- 
tor with very little information useful in conducting an evaluation. 



unintended effects. As Scriven 



pu^s the matter "its risky to htnir 
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What the evaliiator needs are statements of speeifie goaJs and objec- 
tives. It is usually over the speciKc goals and olyeetives of a course 
or program that teachers and writing prograin directors disagree, 
even though they may agree on the general goal^Tand objt'ctives. 

For an evaluation to he useful, it seems to us ^hat teachers and 
writing program directors must agree in advance of the actual eval- 
uation on the specific goals or objectives of the writing program and 
of the courses included in that program, especially as those goals 
and objectives immediately aftect students enn)lled in the program. 
Getting agreement is, however, frecjuently difficult. Indeed, a re- 
cent national survey of teachers' and directors' perceptitms of the 
goals of freshman writing programs indicates nmsiderable disagree- 
ment between the two groups on the matter of goals;'*' and that sur- 
vey combined with a survey of the writing college graduates'*"* iif^^*' 
dicates that neither writing program directors nor teachers may 
perceive the same goals for college writing programs as people? vyho 
have graduated from a)llege. While meetings with the teachers/and 
the writing program director may help the evaluator identify some 
of the sj>ecifte gtmls and oljjectives. it is unlikely the gnmp will ar- 
rive at consensus, except through the influence of dominant and 
dominating pers«)nalities. Weinberg explains the pr blem and rec- 
ommends a solution; 

Now coinniittees in my view can no^mfttx*, pnuliicc wisdom than they 
can design a camel. The atmosphere of a committee is too rompctitive. 
too vcrh^i. Wisdom is a vcr:> personal kimi of thing: it ilourishes best when 
a single mind thinks (jnietly and consistently — more (jiiietly and consis- 
tently than is {xjssible when one is engaged in the rongh-and-tuinhle of 
c^ommilteeship with its often tendentious ami personal exchanges. Thus. I 
have tell that some ot the most troiihlesome (jiiestions (m^ht to he thought 
through My individuals who would then set thoughts down in essays. Out 
of many inc h essays, written f)y different pewple. coidd eome. if not clar- 
ity and guidance, at least a eonnnon language in which to ronduct the 
discourse."' 

Such written responses ctndd i)e analyzed for content."" for different 
statements of specific goals and objectives. I hese statements could 
then be used in connection with the Delphi technitpje/"* The i^el- 
phi techni(jue involves collecting stat(»nunts. such as those sug- 
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gested by VVeiiibern, siiyrriiariziiiK thepi. and tlieii circulating the 
summary The uroup. ilT this case the teachers and the writing pro- 
gram director is then asked to re-evahiate their positions on goals 
and objectives in light of the summary dot^ument and then to sub- 
mit another statement. These new statements are again summa- 
rized, the summai]y circulated, and revised statements again col- 
lected. The pntt^s continues until consensus is reached. 

The goals aiil oi)j^tives for student peribrmance. of course, can 
be of many kin J.s. And whether the Delphi techiiicpie or some other 
method is used to identify them, the evaluator— as well as the teach- 
ers and the writing program director— must be able to group or 
classify those goals and ol)jectives. One classification system that 
frecjuently appears in the literature ( n educational goals and objec- 
tives identifies cx)gnitive,*^ affectiv*-, " and psychomotor goals and 
objectives. The usefulness of these three divisions may be that they 
allow goals and olyectives to be 'classified according to a a)nsistent 
framework which uses a vwabe ary most teachers and writing pro- 
gram administrators are alrea^!y familiar with. In addition, it allows 
an evaluator or a group of ivrsons associated with an evaluafrbhrto 
determine what might l)e called the ^'balance^^ among the diflerent 
types of goals and objectives. Although there arc inherent weak- 
nesses in any taxonomy compcsition programs and courses proba- 
bly have goals which address tlie cognitive and affective domains of 
the learner And the more general of these goals are often stated in 
one form or another — in colU'ge catalogues, in writing program 
policy statements, and in coi/rse syllabi, l^-ss obvious, and less 
prevalent, are psychomotor guiils, for most college programs simply 
assume that students have a)iiimand of the psychomotor skills nec- 
essary for them to perform adetiuately 

Other taxonomies can also I)e used to classif y the goals and oi)jec- 
tives of college writing coursles and programs that bear on student 
performance. As Yow pointyout/* Gagnes categori^^s of learning- 
chaining (i.e., stimulus-rt/sponse), verf)al association, multiple 
discrimination, concept learning, principle learning, and problem 
solving— can be used to classify the goals and objectives of an edu- 
cational program.'* The/three higher level categoiies— c^oncept 
learning, principle learning, and problem solving— seem particu- 
larly relevant to many /bf the goals and objectives of writing pro- 
grams and courses, Tl^be same goals and objectives might "Uo be 
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classified according to Vygotskys stages oi concept ronnation.'^ 
Vygotsky distinguishes among three such stages; (1) the putting "to- 
gether a number of objects in an unorfianized confierie^i, or *heap/ 
in order to solve a problem that we adults w'ould nornialU' solve by 
forming a new concept ' (p. 59); (2) **thinkinp, in complexes' during 
which "individual objects are united . . . not only by . . . subjective 
impressions but also by hands actualhj existing between these oh- 
jects ' (p. 61); and (3) using true concepts (pp. 61-118). It seems to us 
that manv instructional activities — such as the diflerent kinds of 
sentence combining exercises — might be placed at the diflerent 
"cognitive ' levels suggested by Vygotsky s stages of concept forma- 
tion. Piaget s stages of mental development might also be used as 
the basis of a taxtmomy of goab* and objectives Ibr writing programs 
and courses. Piaget distinguishes among four such stages — the 
sensori-motor, tlie pre-operationaK the concrete operational, and 
the formal -operations stage. ''^ Also potentially useful is the well- 
known distinction (Chomsky makes between competence and per- 
formance.^ This distinction is. in fact, applied in a modified form to 
performative goals and objectives by the contributors to The Nature 
and Measurement of Com)>eten€ij in English, 

We have argued above that evaluation studies typically take the 
written texts of students as the primary evidence of the ettects of 
writing programs. The assumptions underlying this practice are 
three: (1) that students* written texts collected under testing condi- 
tions are adequate reflections of writing cimipctency (2) that legiti- 
mate inferences about writing course and program eflectiveness can 
be made on the basis of student texts, and (3) that writing programs 
only affect student products. Kvaluators have come b> this practice 
(juite naturally: the country's writing assessment experts — the Na- 
tional Assessment of Educational Progress and the Kducational Test- 
ing Service — use the written texts of students to make inferences 
about writing competency and program effectiveness."'' In addition, 
it is a practice used by most classroom teachers of writing to judge 
both the competence of student writers and to gauge the effective- 
ness of writing instruction. Most evaluation studies — including the 
four evaluation studies we reviewed in the previous chapter — base 
most of their conclusions on students' written products. Implicit in 
all of these evaluations — from individual classrooms to national at- 
tempts at writing evaluation — is the assumption that written prod- 
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nets represc lit t\\v "bottom line" in assessing stiicltMit pfHormaiicr 
ami writing course eflcctiveiu'ss. 

Ill larne-scalc evahiatioii.s^ two kinds of sttidnit products have 
been assessed: student responses on standardized tests of writing- 
related skills and actual writing samples. Standardized tests are 
used in indirect assessments of'wptiiiK, i*»d writing samples are the 
basis of direct assessments. There are strengths and weaknesses in 
both forms of assessment. Pro|K>nents of indirect assessments ar^ue 
that standardized instruments are more reliable than direct assess- 
ments and less expensive to administer (Critics of indirect assess- 
ments, on the other hand, (juestioii the validity of such instruments 
as measures of writing abilitx since examinees usually write no more 
than their names. Proponents of direct assessments ar^ue that the 
only valid measure of writing competency is a writiim sample, in 
spite of the fact that they are relatively expensive and ditfienit to 
score and that they predict success in writing courses no better than 
indirect assessments.''* 

But validity can hv questitmed in direct assessment as well as in- 
direct assessment, bi recent times the issues of validity in direct as- 
sessment has usually l)een raised with respwt to the number and 
kinds of writing samples recpiired to assess competency.*'' Further 
({uestions can be raised about the kinds of cognitive abilities direct 
assessments measure and the writing tasks themselves. The NAEF, 
for example, once asked students to write in response to the Ibllow- 
iuK (piestions: "What would it be like to be a goldfish? Or an air- 
plane? Or a horse? Or a tree? Or any other thiiiK?"*'' What is beiiin 
measured with this assignment? The students' ability to use their 
imaginations? To communicate with a "real" audience? To show 
their knowledge of goldfish, horses, or whatever? 

Perhaps a more critical (juestion bearing on the validity of direct 
assessments is their inability to assess writing prcKcsses adecjuately, 
bideed. most direct assessments re(iuire that students write in very 
limited time span on an impromptu topic, a topic about which some 
of the (»xaiiiiiiees ma> have a >^reat deal of knowledge and others 
may not. This procedure runs directly counter to what we know 
about how experienced writers compose.'*'^ If writers do not have 
command of the requisite prior knowledge to write about a subject, 
then they must have the time to n^Mierate ideas on that topic. In any 
case, most direct assessments do not provide adeipiate indications 
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of skills in invfiitioii. Kurtheriiuire. direct assessiiiei^t does not al- 
low for substantial revision, fnr adjnstiiiK a text to the needs of a 
"real" audience if one were provided. 

FolIowiuK the lead of iiunierous researehers who have studied 
composinK processes, many writing teachers now believe they fol- 
low a process-oriented curriciiluin.*'* Both indirect and direct assess- 
ments of writing tell us very little about how students' coinposiiiK 
pnxesses mi^ht change as the result of instruction, about whether a 
studv»nt who habitually stops thinkiuK about a paper at the end of a 
single draft minht be williiiK to take a paper throuKh multiple drafts. 

Thus products alone may not give an adecjuate picture of writing 
competency. Nor can products alone give adecjuate intbrinati(ui 
about the efl'ects of writing programs. While writing programs have 
traditionally sought to improve the (juality of written products, pro- 
grams produce other im|M)rtant eflects as well For example. s(une 
writing programs and many writing teachers have as a goal the de- 
velopment of cognitive abilities that allow students to perform bet- 
ter in courses in other disciplines. Indeed, many writing programs 
claim to teach intellectual pnKesses— such as inducticm, deductwm. 
and clasxifrratKin^which are fimdamrntal to all academic disci- 
plines and are believed t(» iittect the perlbrnuuices of people after 
college. S(Mne students, both during and after writing instruction, 
may be able to »ise skills in reading and thinking without employing 
them in writing, thus making direct assessments ot writing perhaps 
!iiappropriat<» indicators (if certain intended efl'ects of a writing pro- 
gram. In addition to devehiping ccmtrol over ccunposing and other 
cognitive processes, composition programs also sometimes aim 
to produce changes in attitudes which are not reflected in written 
products during the term of the c(mrse. For at least some students, a 
change in attitude toward writing may be just as important as im- 
provement ill written products. Students who fear writing have 
hvvn shown to avoid courses and jobs that recjuire writing.'^ More- 
over, writing programs often seek to increase students' appreciaticm 
iA written lamr ige as a medium of comimmicatioii and means for 
self-expressi(u: hese eflects may not only be inaccessible through 
assessments of written products but may not even occur until some- 
time after students have completed their formal c(nirses in writing, 

Vnintended Effects Not only must program evaluators be c(m- 
cerned with identifying the intended eflects oi writing programs 
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and coursfs, hut thrv niiist also takf into atrount the uninttMuled 
efffcts. Om of the hest examples of unintended efietts is the nega- 
tive attitude toward writing that many stiidt'nts brinn to the writing 
classroom. Many a)lleKe teachers attribute these attitudes to error- 
oriented instruction in previous courses dealing with writing re- 
gardless of discipline. Other unintended eflects may he the devel- 
opment of misconceptions about writing, both as product and as 
process. Some of these misconceptions center on writing processes 
— such as writing is easy for good writers or that good writers get it 
right the first time. Others center on products — such as every para- 
graph should start with a topic sentence or that good writing always 
reads like an essay by George Onvell or K. B. White, Not all unin- 
tended eflects concern students. For example, the recjuired use of a 
common syllabus in a particular writing course may cause teachers 
to alter their teaching practices to such an extent that they feel they 
h)se their eflectiveness. The number of papers that teachers are re- 
cjuired to grade according to program policy during a term may ad- 
versely aflt'ct morale in a department. Writing programs have other 
unintended eflects beyond their immediate confines. Because En- 
glish departments have traditionally taught most writing courses, 
responsibility for teaching writing has been shunned in other de- 
partments, a situation which i'^ changing in many institutions. Fi- 
nally, writing programs — for good or bad — have had some responsi- 
bility in shaping attitudes toward language and writing held by the 
public at large. 

III. 3. Interactions Amonfi the Five Components 

As we intimated in previous sections, writing program evaluation 
nuist necessarily be concerned with the eflects of writin:4 programs. 
Which eflects are important depends on the way the evalviator 
chooses to view writing program and what (piestions need to be 
answered. As we have seen, writing program evaluators can choose, 
for example, to iMuit the scope of an evaluation to an examination of 
the eflects of a particular instructional method on students' written 
products. Such an evaluation entails a numistic view of writing pro- 
grams, with respect both to eflects and to possible causes. That is to 
say, such an evaluation is unidimensional in nature, while the writ* 
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ing program hvin^ evaluated is inultidiinensional. consistiiiK at least 
of a cultural context, an institution context, program structure and 
administration, a turriculmn. as well as a pedagogy. 

To evaluate a writing program unidimensioiiallv is to engage in an 
activity similar to the examination of the elephant i)y the six blind 
men. If the evaluator examines only the "trunk" of a writing pro- 
gram, the interdependence of all the parts will hv missed. What- 
ever the elephant does, it does because its several parts work in eon- 
cert with one another in a particular envin)nment. So t(K) with 
writing programs: no single component can he singled out f()r exami- 
nation if the evaluation truly seeks to understand the cause or causes 
ol this or that effect. Evaluators nmst be aware of the ways the five 
comijonents can combine, or interact, to cause certain efik-ts. 

If we consider all the possible cond)inations of the five compo- 
nents, there result 26 possible sets of interactions which deserve 
the evaluators attention. In the most comprehensive evaluation, 
the evaluator would look for the causes of particular effects in the 
complex interacticms among the cultural context, the institutional 
context program structure and administration, the curriculum, and 
instructional methods.. In less comprehensive evaluations, one .)r 
more of the 25 lesser interactions would be of importance: 

1. cultural and swial context with institutitmal context, 

2. cultural and social c-ontext with program structure and ad- 
mini$trati(m, 

3. cultural and social context with curricidum. 

4. cultural and social ctmtext with instructi«)n. 

5. institutional context with program structure and iuhninis- 
t rat ion. 

6. institutitmal context with curriculum, 

7. institutional context with instruction. 

fi. program structure and administration with curriculum. 
9. program structure and administration with instruction. 

10. curriculum with instruction, 

11. cultural and social context with institutional context with pro- 
gram structure and administration, 

12. cultural and social c«)ntext with institutitmal a)ntext with 
curriculum. 

13. cultural and swial context with institutional c-ontext with 
instruction, 
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•4. cultural and stK-ial context with program structure and ad- 
ministration with curriculum, 

15. cultural and stK'ial context with program structure and ad- 
ministration with instruction, 

16. cultural and swial context with curriculum with instruction, 

17. institutitmal context with program structure and administra- 
tion with curriculum, 

18. institutional context with program structure and administra- 
tion with instruction, 

19. institutional context with curriculum with instruction, 

20. program structure and achninistration with curriculum with 
instruction, 

21. cultural and stKial context with institutional context with pro- 
jimm structure and administration with curriculum, 

22. cultural and social context with institutional context with pro- 
gram structure and administration with instruction, 

23. cultural and social context with institutti)nal context with cur- 
riculum with instruction, 

24. cultural and s(K'ial context with program structure and ad- 
ministration with curriculum with instruction, 

25. and institutional context with program structure and admin- 
istration with curriculum with instruction. 

These interactions are vital to understanding the effects of a particu- 
lar program. In any fiiven interaction, the evaluator can look for the 
effects of one c*omponent on another or of two, three, or four compo- 
nents on another component. In the next chapter, we will discuss 
how these components and the interactions anion); them lead to eval* 
nation questions which require the use of'nuiltiple methodologies. 
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THROUGHOUT THE prkvious thkek chapi ehs. wk have su(;- 
Rested that the complexity of writiiin pro|i;rains has done inncli to 
limit the development of adequate evaluative procedures and nietli- 
ods. Yet the sheer coMpIexity of the thiny evaluated is not the only 
reason the art of writinii; program evaluation remains in its infancy. 
The nature of evaluation itself raises many (|uestions and issues that 
should Iw reexamined in every sij^nificant evaluation. We have in 
mind those evaluaticms which have the potential to aHect the lives of 
others, the kinds of serious evaliiati(ms in which evaluaticm itself is 
often an issue. Usually these questions focus on the validity of the 
evaluation. Such (piestions certainly arise when evaluation is mis- 
used covertly in the ways outlined by Suchman — as a way to "eye- 
wash" in order to call attention to only the Hoiyd aspects of an oth- 
erwise poor program; as a way to "whk^wash * in order to hide a 
program s failure; as a way to "submarine" a program regardless of 
its worth; as a way to create the appearance or "posture" of objec- 
tivity; and as a way of postponing administrative action under the 
pretence of insuificient facts.' Yet ({uestiims of validity have been 
raised about even the niost conscientiously conducted evaluations, 
whether of writing programs^ or of social and educational programs 
generally. * 

As we indicated in chapter 1. our central concern in the present 
monograph is with validity in writing program evaluation. Our dis- 
cussion in chapter 2 of the Northern Iowa. San Diego» Miami, and 
Texas studies litK Used on matters which bear on the validity of those 
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evaluations. And in our discussion ol'tlie quantitative model ofwrit- 
iiiK program evaluation, we tried to point out the inlienMit weak- 
nesses and limitations that n»duee tlie validity of results generated. 
In the third chapter, we presented the major components of a theo- 
retical f'ranu^work tor conducting valid evaluations. We argued that 
this framework am)mmodates or amnnits for the five necessary 
conjponents of college writing program evaluation: the cultural anci 
social context of the program, its institutional context, its structtirc 
and administration, its curriculum, and its pedagogy At the end of 
chapter 3. we suggested that writing program evaluators most l(M)k 
to complex sets of interactions among those five components in or- 
dvr to develop a ct)mpn»hensive view of a program and its efiects. 
lw)th intended and unintended. The first three chapters of the pres- 
ent monograph thus define the ''territory'* aflecting validity in writ- 
ing program evaluations. 

Validity is the .entral issue addressed in the present chapter as 
well. Validity depends on the appropriateness of the cvaluaticm to 
the nature of the thing evaluated. In the following section we ad- 
dress briefly the development of and the need for two distinctive 
approaches to evaluation. In the second section, we point out how 
changes in writing programs and in conceptions of writing rccjuire 
new evaluation prcKedures and materials. In the third section. We 
suggest directions which research must take in order to develop 
valid procedures and materials Ibr evaluating writing programs. 

IN' 1. Quantitative and Qualitative Methods in Evaluation 
Research 

Whatever virtues or vices may be discussed with respect to eval- 
uation, the issue of validity is always present. Questions about the 
validity of a writing program evaluation are often raised because the 
data examined and the analytic methods and paradigms employed 
are often distrusted by persons who have a professional interest in 
the teaching of college writing. Fart of this distrust stems, of course, 
from the backgromids of college faculty who are chargi»d with the 
teaching of writing. Such persons often have strong literature back- 
grounds • which better prepare them to analyze single written texts 
tlian to analyze enormous amounts of data collected under nM)re or 



ERLC 



80 



68 Kvaiuat^^^ (Joiie^c W'nUiiu Frograins 



less a)ntn)llt*(l eoiulitions. Thr research method familiar to most 
teachers of writings the inethoil of the literary scholar whi) con- 
Ironts a tcxf nnc-on-tme. and from such a one-on-one meetinu dis- 
covers certain truths about life, the human ccmdition. and art. As 
Kisner has recently argued, there is much to he said for "artistic ap- 
proaches to ri^search." approaches which try. as literary scholars 
nnist do, "to locate the Keuera! in the particular." Kisner continues: 
"(such approaches] attempt to shed light on what is uni(|ue iu time 
and space wfiile at the same time conveying insights that excved the 
limits of the situation in which they emerge. This is precisely what 
Aristotle meant when he naid that Toetry* was truer than history/'^ 
Yet while there is much value in employing evaluatitm methods 
comparable to those of the literary scholar, it is often impossible to 
conduct a comprehensixe evaluation of a large program or even a 
single course using exclusively those methods. Kvalua^'ions of writ- 
ing programs camiot c*oneern themselves with individual pi^iibr- 
mances onl\'. pertbrmances which may or ina> not b(/generalizable; 
they must Uv c«)ncerned with the performances o/ large groups of 
students and often with large groups ol teachers, a circumstance 
which may demand a combination of several m(*thodologies. 

There are good reasons, however, lor distrusj'ing the ({uantitative 
aspects of evuhiatiou. lor as we pointed out in /nir review ol,tlie four 
studies, ({uuntitative methods do not necessarily produce usehil or 
valid results or even reliable results. Thus, a central issue is the ex- 
tent to which evaluation should be regarded as a science. Some re- 
searchers, for example*, have maintained that (jualitative methotls 
and models are highly (|uestionable. as have ('ampbell and Stanley 
in their classic work on research design. Addressing one particular 
(pialit:itive method, ('ampbell and Stanley write that the case study 
approach typifies "the error of misplaced precision" and that '*It 
seems welbnigh unethical at present to allow, as theses or disser- 
tations in education, case studies."'* \'ery recently, the quantita- 
tive bias of (Jampbell and Stanley has been reiterated by Hossi and 
Wright: "There is almost universal agreement among exaluation re- 
searchers that the randomized controlled experiment is the ideal 
model for evaluating the ettectiveness of public policy. If there is a 
Bible for evaluation, the Scriptures have been written by Oampbell 
and Stanley."' Yet even (^ampbell." as well as other such noted 
({uantitative researchers as Oroiibach, * now apparently finds (juali- 



er|c 



81 



AcctHniiHxJutinu (ionU*xt aiul Oliant^i* 



tativi' methods mow accvptahle. Spi'ukinn tlinvtly to the tnnaTHs 
of the evahiator, Kenimis aildresses the (iuantitativ(»/c|ualitativf dis- 
tinction in the ibllowin^ way; 

Too often evaUuitors itssiime that tlieir tUvuis want tlieiu to lollect a 
b<Kly of ' objective" test data so that the dionls tan make diTisions alniiit 
educational prourains: it is assuni(>ci that these* decisions follow hy ini- 
pheation iVon) tlu* evaluation data. In their sophistication, the (^valuators 
may lor«et thai the data they uive to their clients create only a hare 
skeleton of the many-sided reality ol the program. It is cold and imyield- 
Hiuto the technically inexpert eye: it dcH»s not ade({uately render the liv- 
inU reality of people, events, and issues in day-lo-da\ prourani operaticm. 
Furthermore, the reality it does create, that of Outnnnes achieved and 
not achieved, preseni"d in scores, tables, and graphs, may actually mis- 
/m/ program personnel insotar as it leads them to value* those things that 
the (Avalualor can measure at the exjwnse nl those aspects ot th" situation 
IcM) elusive to he captured hy his measurements.^" 

One of the fundamental differences between (piantitative and 
({nulitative methods is th<' n)le of the researclier: in a (piantitatiVe 
study, the researcher stanu.s :<part from the object of stndy. hut in a 
(piuhtative study, the researcluT f:.f;eomes sometiiing of a partici- 
pant, intent on beinn able to represent the coniplexity of the activi- 
ties under study. 

But this diflerence between the roles evaluators play in quanti- 
tative and (pialitative appnmches to evaluation is not to l)e taken 
lightly as a simple diH'erenee in procedure. Quantitative* and (pialita- 
tive methods embody fundamentally different theories about both 
the nature of reality and about the nature of knowledjie and knowl- 
edge accpiisitions. Fatton. one of the better known authorities on 
evaluation, contrasts the two methods and indicates the historical 
evolution ()f the two often competing paradif^ms: 

Kvaluation research is dominated hy the largely un({<iestioned. natural 
science paradigm of hypothetico-deductivi* mcthudolouy. 'fhis iloniinant 
paradi]ii;m a.«>:umes quantitative measurenu^it. experimental desi^n, and 
multivariate, parametric statistical analysis lo be the epitome of '^ood" 
science. This basic model for conducting i^valualion research comes from 
the tradition of experinu'ntation in auricidutre. which Uuve us many of 
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the basil* statistical and experimental techiu(|ues most widely used in 
evaluation research. . . . 

By way of contrast, the alternative to the dominant . . . paradigm is 
derived irpm the tradition of anthropological Held studies. Usin^ the 
techniques of open-ended interviewing and personal observation, ihe al- 
ternative paradigm relies on (qualitative data, holistic analysis, and de- 
tailed description derived from close a)ntact with the tar^vts of study. 
The . . . natural sciciicc paradigm aiins it prediction of social pluMioin- 
ena; the holistic-inductixe, anthroi^olo^ieal paradigm aims at under- 
standin)i; of social phenomena." 

The central (juestion is. ot course, whether evaluation research 
ounhJ to represent itself through its methodology as a hard science 
after the manner of chemistry or physics or as scnnethiu); closer to 
research in the humanities or the social sciences. In educational re- 
search in neneraP^ and in evaluation research in particular.'* there 
seems to be a trend toward selecting? anions a variety ofmethodoK)- 
Kies, in part because neither the natural science, social science, nor 
humanities approach is necessarily the correct one.** As Patton puts 
the matter. 'The debate and competition between paradipns is 
beinn replaced by a new paradii^ni — a paradifim of choices. The 
paradif^m of c!u)ices reco|i;ni/es that diflerent methods are appropri- 
ate ior difierent situations/' 



IV 2. Chanf^iufi Perceptions of Writinf;. Ever-chanfiiufi 
Writirifi Prograim 

The changes in assumptions about proj^ram evaluation that we 
have described reflect bn)ad changes in higher education in the years 
following World War II. The extension of educational opportiuiitie.s 
to persons who traditionally had not attended college changed higher 
education from an essentially elitist to a fundamentally egalitarian 
institution. Most aHected by these dramatic changes were programs 
teaching basic skills, especially introductory mathematics and writ- 
ing programs. College and universities were simply not prepared to 
instruct the diBerent kinds of students that appeared in large num. 
bers on their campuses. 
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The channiiiK student populations brouKht established methods 
i)f teaching writiuK i»to question. In 1961. Kitzhaher deiicribed two 
predominant ways of teaching writing— the hterature approach and 
the rhetorical approach."' By the end of the 195()s. these approaches 
to the teaching of writing were recognized by some faculty mem- 
bers as inadequate. In 1959, Warner Rice called for EngHsh depart- 
ments to stop teaching what they did not know how to teach and to 
return to the teaching of literature.'" Further changes in American 
society during the 1960s accelerated the decline of the traditional 
approaches. The decade of the 1960s was the decade of liberal ide- 
als: authority was challenged, requirements were dropped, and the 
written word was no longer sacrosanct. As Brent recollects: 

Students were turning; in (tillages instead (if essa\ s in many cdinposi- 
tinn ccnirsrs In (me wuirsc in American literature taugiit liy a teachiiig- 
assistant nffiLC-niate nf mim\ the students didn't write papers at all. They 
baked bread. KvidentK; somewhere alnuK the line. Thvireau nrinhieed a 
recipe for bread, and the students baked ac^-urdingly Thvy would line up 
outside the oifiet» with their loaves and approach my friend s desk where 
he had a paper plate and a bi^ knife. He could cut a hunk ofi'each loiii. 
chew it for fifteen seconds or so. progressively more slowly and slowly as 
his expressum became more and more sagacious, and finally he would 
look at the student and say. "B plus" or "A ininus."''* 

Ironically, the saitie liberal ideals that led in some places to the al)ol- 
ishment of freshnuip English as a required course eventually led to 
the reintroduction of writing rr^o the a)llege curriculum in a much 
more substantial way than ever before. College and universities 
nationwide relaxed their entrance requirements or madi' provisions 
for special admissions, giving access to higher educatiim to a large 
segment of the population that previously had not attended college. 
The problem was not confined to institutions with open-admissions 
policies. Established universities, such as the University ofCahfor- 
nia at Berkeley, found that a majority of students from middle-class 
bacVgrounds could not write "college-level" prose. 

During the 1960s, another major change occurred in the soc^ial 
context that will likely alter substantially the nature of writing pro- 
grams. This change was not obvious to us until v/e conducted a strati- 
fied survey of the writing of college-trained people in the work force. 
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i^though a groat doal of anmlotal evidence has l)tni colliTtod that 
indicates writing is an important skill on the job. and several spe- 
cialized surveys of writing in business and industry have been con- 
ducted, ua one to our knowledge had attempted to survey a cross 
section of college graduates. We obtained a sample of 20() individu- 
als which adequately fit the U.S. Department of Labor statistics on 
the diiftributioii oi college-educated people in the American work 
force. Over three-fourths of these wllege-trained people are em- 
ployed in technical, professional, and managerial occupations, and 
fully half of all college -educated people are in technical and profes- 
sional ocx'upations. For all respondents the average total work time 
spent writing was 23. 1 percent, or over one day in a five-day week. 
Nearly three-fourths of the people sampled claimed to spend 10 per 
cent or more of their work time writing. Only four people claimed 
never to write while on the job. All of the people in technical and 
professional occupations wrote on the job. spending on the average 
29 percent of total work time writing — a figure higher than for any 
other occupational group. What we found out about writing on the 
job nms directly t*ounter to the views of many in academe about the 
importance of writing after college, a view still articulated by l)oth 
faculty and students and accepted as fact by the CJom mission on Kn- 
glish in 1966."^' Wri^'ng is an important and frequently used skill 
across all major cate^ »ries of occupations that college graduates are 
trained to enter. We are suggesting that it was not coincidental that 
the literacy crisis occurred at a time when many colleges and uni- 
versities were reducing v: bolishing their writing programs while 
the jobs that their graduates obtained required both riiore writing 
and more miplex and diverse kinds of writing. 

From the perspecl ve of the early 1980s, we can see two major 
developments stemnnng from the inadequacy of the traditional 
approaches ty' the teaching of writing and the public outcry over 
the **liter^tfy tTisis/* Both develooments are related, and both are 
founded upon relationships between writing and thought. The 
more general development is the growing emphasis on processes in 
writing; the more specific development is the writing-across-the- 
curriculum movement/* a development which seems to have re- 
sulted as much from pressure brought to- bear on Fnglish depart- 
ments from the outside as from any measurable commitment to the 
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concept from inside English departments. Ainong the more visible 
manitestations of'these two developments have been the followinu; 

1. Writing is no longer tatight only in the English department or 
only in the freshman year. 

2. Writing courses are recognized to have special needs, espe- 
\ cially special training tor their instructors. 

3. Students write diflerent types of writing for ditterent audi- 
ences and purposes instead of themes. 

4. The instructors role has changed from strictly that of an eval- 
uator to that of a coach. 

5. Writing is taught by having students write instead of by lec- 
ture or discussion of readings. 

6. Writing is not defined as a static product. 

The movement toward teaching writing as a process goes beycmd 
merely having students write multiple drafts. It reinterprets the 
functi(m of writing in the overall curriculum. Writing becomes a 
way of discovery. Students write not oidy to report what they know 
but also to discover coimections of which they were not previously 
aware. Students can evaluate their ideas when they write because 
the ideas become explicit. Moreover, writing forces students to be- 
come active learners of a subject, to participate in the vital discus- 
sions of a discipline. 

As we have pointed out in our discussion ofthe(juantitative model 
the change in the nature of writing courses also called into (juestion 
prevailingways of evaluating writing programs. The British Sch(H)ls 
Council recognized this inadetjuacy when it conunissi(med a stt)dy 
of the writing development of children in British schools, not oiv the 
l)asis of ({uantitative measures such as clause length but in terms of 
the functi(ms of what they wrote. The work of the Schools Council 
. research team, James Britton and his University of Londoi^ col- 
leagues, came to the conclusion that the great majority of texts writ- 
ten by schoolchildren were lifeless eBbrts addressed to the teacher 
as examiner. When the interests of writing teachers and research- 
ers turn to questions of process and fifjiction. the measures devel- 
oped for pretest-posttest evaluatioia,i)ecome inappropriate. At the 
same time, thi' expert-opinion ap|<roach to evaluation can be of lit- 
tle help berause it lacks the theoretical underpinnings necessary to 
address (juestions of process and ftmction. Significantly. Britton and 
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his colleagues had to develop a theory of discourse in order to make 
judprnents about the t'uiictioiis of* writing in British schools. The 
changes in tlie nature of program evaluation, in writing programs, 
and in research in writing underscore the need for developing alter- 
native? models for writing program evaluation. 

1\' 3. Questiom for Evaluators 

Pattern's notion of a "paradigm of choices ' is a useful one l)ecause 
the complexity of writing programs, we believe, precludes the use 
of a single approach in evaluation. Approaches to evaluation are, in 
effect, models of evaluation, such as the model we c»xtracted from 
the four evaluaticm studies reviewed in chapter 2. The number of 
specific approaches to evaluation found in the literature is large, al- 
though none s'pecifically addresses the problems of writing program 
evaluation. From the standpoint of the evaluator of writing pro- 
grams, the emergetu'v of a "paradigm of choices" is fortuitous. Such 
a paradigm may lead to a synthesis of world \ iews which will allow 
much greater latitude in deciding which kinds of data can be validly 
used in writing program evaluations and much greater latitude in 
tKe inteq)retati(m of those data. This is not to say that a "paradigm 
-of choices" reduces the mimber of (juestions about the validity of 
materials or procedures. To the contrary, such a paradigm increases 
exponentially that number. While a "paradigm of choices" incTehses' 
the options available to the evaluator, it does not eliminate the v*s- 
sential and fundamental diflereiices in world \iew that the cfioices 
represent. 

Knowledge of and skill in using (pialitative approaches may very 
well extend our vision of writing program evaluation beyond its 
present moorings in product-(;riented. pretest-posttest definitions 
of program effectiveness to the contexts of programs and the pro- 
cesses of learning and composing, but at the same time it foices us 
to decide which aspects of context, which aspects of learning, and 
which «ispects of composing are worth the considerable effort and 
cost demanded by close oi)sei vatioii. But it is only tiirough a plural- 
ism of approaches that evaluators will probably ever be able to ad- 
dress the complex sets of interactions among the ccmiponents of 
writing programs posited at the end of chapter 3 or will probably 
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vwr be able tn hmnnr srnsitivr tn the rapid ever-chaiiKinn iiatiiiv 
nl the thiuK evahiated. 

As enlU*K<^* and university writing prn);ranis cnntinue tu change in 
response to siKietv *s demands lor a HtiTate popiilat i* and in response* 
to an ever-^rowin^ hody ol knowledge ahoiit botli the products and 
processes of writing, evaluators will have to addn*ss iucrtMsioKly 
complex questions and issin»s» and they will have to devrhip per- 
haps equally complex evaluations and procedures to address such 
i{uestions and issues. The interactions anions the five compinients 
specified at the end of chapter 3 can serve as a heuristic lor identtly- 
in^ som(* of these questions. 

One category ol questions focuses on the writing program itself 
For example, the interaction anions social context, institutional con- 
text, and program structure mi^ht produce a question such as "\\ hat 
i.s the status of the academic unit responsible lor developmental or 
remedial instruction?' 'I'his question woidd take into account the at- 
titude of the siK'iety at lar^e toward the studiMits in the program, 
the institutional conunitment to the echication ol those students, 
and the organization ol a writing program that distinguishes be- 
tween developmental and nondevehipmental students. A questiim 
such as "How successlul are graduates of a developmental writing 
program in subsequent courses and in the world of work?" wnuld 
necessarily iovnlve all fivi» components. 

A second catt^^ory of qui^stions hnHises oii evaluation. We will 
Kroup these questions according to the lour constituencies olWrit- 
inK programs identified in chapter 3: the snciety, the institution, thi* 
program includitiK its teacher and administrators, and the student. 

One set of questions concerns the efllects of writing programs on 
society. These questions might include: 

1. l>oes the writing program atl'ect the value its students place on 
ihv written language once they leave college? 

2. Does the writing program, through its students and graduates^ 
make* the public sensitive to abuses of language? 

3. Does the writing program help make the public aware of dif- 
ferent usesof hmguag(*s for ditlerent purposes? 

4. Does the writing program aflect the social and economic status 
of its students and graduates? 

5. Does the writing program send students into the world better 
able to adapt the pnKt^sses and pnxlucts of writing to novc»l situations? 
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6, Does the ^vritiIl^ prouraiii foster the (levelopiiunit ofeounitive 
skills that have uppheatioii heyond writiiiu? 

7. Does the writioK prouraiii eontrihiite siibstaiitiallv to the oper- 
ation of a (lenuKTutie s<K»iety. a soeietv whiefi is predicated on the 
assninption of a hterate popnlare? 

8 Does the writinu prourani produce students and uraduates \vh(j 
write in order to make sense of their experiences as human heinus? 

A second set of (juestions miuht tm us ofi the cHects of a prourain 
on the institution which houses it. These (juestions ini^ht inchide: 

1. Arc students capable of writinu intelliuently and clearly ah(mt 
•specialised topics in disciphnes other than Kn^lish':' 

2. Are improvensents in writinu i'.'hility lost alter students leav(» 
the compositicMi pro^ramy 

3. How does the writinu prouram influence aduussions recjuirc*- 
ments at a particular institutiony 

4. Does a writing program affect the munher of deurees awarded 
at a particular institution? 

5 To wliat extent does the si/e of a ^v^itin^ prouram affect xW 
lunnher of other ( (Mirses ottered? 

6. flow does a writing program atlect the makeup of the faculty at 
a particular instituticm? 

7. To what extent does a writinu proyrain pr(nnote ccnnnunuca- 
ti(Mi amouK faculty and students in different disciplines on the hasis 
of a commcm interest in written lauKiiaue? 

8 How does the attitude of the (composition teacher toward the 
teachinu (if writing and tovv«iu stmieiits of writinu c<mti ihute to the 
attitudes of students toward the written lanuuaue in uenenil? 

* u)ther set of ({UestioUs ini^ht he dir(rted to interacti(ms of vari- 
ous kinds uuKuiK the c(mip(miM)ts (»f writinu programs. An:onK these 
(piestions are: 

1. How does a writing program attect the attitudes of teachers in 
the prnuram? 

2. How does a writing pro«raii» aHect the professional status of 
teachers in the program? 

3. How does the use of a re(piire(l syllalms affect what and how 
particidar teachers tea(/h? 

4. How do the support services and e(juipinent (such as tvpiuK 
and copyinR services and audiovisual equipment) affect how well a 
teacher can teacli? 
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5. How dovs the cKissroom itself— its si/e, arraiiKiMiuMit, etc, — 
aflfect how well a teacher can teach? 

6. How does the personality of a writing program drrector aflect a 
writing program? 

7. How tliK's the process of administrative decision making aflect 
what teachers and students do in a writing class? 

8. How does the iinitbrmity or lack ot nnitor.iiity iiiHiieiice what 
teachers and students do in a writiiij/. class? 

9. How does the selection of text ir.-Ueriais aflect the nature of 
curriculum? 

A fourth set of questions could tbcus on the eflects of writing pro- 
grams on students. Such (juestions might he: 

1. Do the theoretical underpinnings of a curriculum aflect what 
students learn? 

2. Do diBering methods of delivery aflect what students learn? 

3. Do students* attitudes toward writing change as a icsult of he- 
iiig in a writing course? 

4. Do curricuhnn and instruction aflect students' awareness of 
their own composing processes? 

5. Does the amount of time spent writing aflect how well a stu- 
dent learns to write? 

ft. Does what a student learns in a writing course aflect perfor- 
mance in siihse(|uent courses? 

7. Does the sequence of curricular elements aflect how well a stu- 
dent learns to write? 

8. What etiect does instruction in critical reading have ou the de- 
velopment of writing ahilities? 

Such (|uestions as tliese are not easy on(»s to answer. 'Hiey prohe 
the nature and eflects of writing programs. They address a complex 
and variously defined process called "learning to write. Further- 
more, tli(*y c(msider "learning to write" as a process that extends 
beycmd the classr(K)ni, that writiiiK has important functions for hoth 
individual learners and s(Kiety at large. 

In the face of such complexity, the impulse is to turn away, either 
leaving writing program evaluations to others or concluding that 
somid evaluations of writing programs are impossthle to achieve. 
^Neither alternative is acceptable to (uir profession. By attempting to 
answer such (juestious, we can better come to know what we do, 
how we do it, and why it is important. If answers are possible, they 
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are obtainable only through a pluralistic approach to evaluation that 
acknowledges the history of vmting and the teaching of writing, 
builds on theories of learning and language, and incorporates a vari- 
ety of evaluation methodologies and procedures. 
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Thomas, The Evaluation of Composition Instruction (Inverness CA- 
E'jgepress. 1981). 

5. this project is supported in part by a grant from the Fund for the Im- 
provement of Postsecondary Education. Some of the work of this proj- 
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6. See. for example, Charles R. Cooper and Ia*v Odell. '*lntnKlucti<m." 
in Research on Composing Points of Departure, ed. Cluirles Cooper 
and Lee Odell (Urhana: National Couneil ofTeachers of Kn^lish. U)7h). 
pp. xi-xvii, and Nancy t Sonuiiers. "The Need for Theory in CoiMpo^i- 
tion Kosearch.'* CoUefie Composition and Communication 31 (1979): 
46-49. In addition. Odell. in 'Teachers of ('oniposition and Needed 
Ke^earch in Discourse Theory," Collcfie Composition and Communicu- 
tton 3() (1979): 39-45, arft^ues that theories of disaiurse need »o he vali- 
dated through experimental research involving writers of differcMit ages. 
When such validation in aca^mplished. then it is proper to l(Mik for and 
test pedagogical applications. 

7. Kichard Bradd<Kk, Richard Lloyd-joncs, uid Umell Schoer. Research 
in Written Composition (Champaign. Naticmal C(uuu'il of leachers of 
Knglish, 196.3). 

8. I). (>ord(m Hohinanand .-Xlberl O. W'lecke, Pre-Writinii: The Construc- 
tion and Appheution of Models for Concept Formation in Writinti, V S. 
Olfice of Kducation (^xjperative Research Project iw. 2174 (KasJ l^ns- 
ing: Michigan State l'ni\.. 196-1). See als<j R<)lnnan's "Pre-Writiiig: The 
Stage of Disct)ver> in the Writing Process." Colletie Composition and 
Comnmnication 16(1965): 106-12. 

9. Si»veral writers have addressed one or huih of these issues. Among them 
are four whose work appears in Sentence Comhinint: and the Teachinc 
of U'rtfirig. ed. Donald Daiker. Andrew Kerek. and Max Moreulierg 
(Akron: L & S Books. 1979): John C Melkm. "Issues in the Theory and 
Practice of Sentence Combining: A Twenty-Year Perspective." pp. J- 
38; james L. Kinneavy. "Sentence Combining in a (iomprehensive 
iMmguage Framework," pp. 6()-76; Harold K. Nugent. The Role of 
Old and New Inforuiati(m in Sentence C(nubining/' pp. 201-8; and 
Arthur L. Palacas. "Towards Teaching the I^ogic of Sentence Connec- 
tion,'* pp. 192- 2(K). See also Andrew Kerek. "The C(nnbining PnKrss." 
in Selected Papers from the WHl Texas Writing Research Conference, 
ed. Maxine Hairstoii and Cynthia Selfe (Austin: I'niv. of Texas. 1981). 
pp. 97-115; and Joseph M. Williuins, "Defining (iomplexity.** College 
Kng/iv/i 40 (Jan. 1980): 595-609. 

10. N.AKP reports have bi^en issued regularly since 1970. The most impor- 
tant NAEP documents i>ertaining to the third and latest nuti(mal assess- 
ment of writing are Writing, Achievement, i96*9-79. Results from the 
Third National Writing Assessment, Volume I — IT-year-Olds, Report 
w. 10 W-01 (Den\er: NAKP 1980); WritiniiAchiewment, m9-79. Re- 
sidtsfrom the Third \ational Writing Assessment, Volume II — /'j-Vefir- 
Olds, Report no lO-W-02 /Denver: NAEP 1980); and Writing:. Achieve- 
nieni, 1969-79: Results from the Third Sational Writitiu Assessawnt, 
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Volume II l~9*)enM}ld\. Wvpariua. 10-\V.03 iDnntT: NAKU i'lSO). 
For a ri'vii'w ol'thf iiiatcrials on rradiiig and writinK riliasnl in U)72 
and 1973 and for an evaluation of the assessnuMit itself, svv John C. 
Melhni. SaUonal Assessment ami the Teaching of English (Url)ana: Na- 
tional Coiint'il of Teachers of Kn^lish. 1975), Mellon also (»xanjines the 
NAEF's Writinti Mechanics: 1969-1974 in "Romul Two of the Natitnial 
Assessment~Inten)retinu the Apparent Decline in WritiuK Ahility: A 
Review. " Research in the Teaching of English 10 (1976): 66- 74. For 
a discussion of njany of the critical issues involved in such lar^e scale 
testing, see Michael Clark. "Contesls and Contexts: Writing and Test- 
iuK in School. " C()//e;;c' English 42 (1980): 217-27. and Paul Olson. A 
View ofPouer: Eour Essays on the Sational Assessment of Educational 
Pmurm (Grand Forks: Vniv. of Nurth Dakota Study Group (ui Kvalua- 
li(n). 1976). 

11. 8 Dec. 1975. p. .58. 

12. Son)(» evidence of tht»se developments in college writing proj^rauis 
appears in Stephen P. W'itte. Paul l\ Meyer. Thomas H Miller, and 
lA»ster FaiKle\. A Sational Sun eif of College and Vniversittj Writing 
Proaram Directors. FIPSK (;rant (;()0WK)5896 Technical Report no. 2 
(Austin: Writing ProKraiu Assessment Office. I'niv. of Texas. 1981). 
KRIC D<K'. no. KD 210 709. Sec also I^wrence Peters. "Writing Across 
the Curriculuiii: Across the U.S. ." mimeograph of a rejxirt detailing the 
results of a naticmal survey e(nulucted hy the National Network of Writ- 
iuK .Across the Curriculum Programs, (ieorjle Mason Tniv.. 1982. 

13. According to Norman Morris. "An Historian's \'iew of K.\aminati(ms." 
in Examinations and English Education, ed. Stephen W'isemar Man- 
chester Maudiester Univ. Pr. 1961). pp. 1-43, the first use of wiiting 
in mass testing (xrurred in Boston in 1845. By 1897. a major reiKirt hy 
J. M. Rice was puhlished in wlucli inferences were drawn al)out the 
ettectivcness of school districts on the hasiJ of tests of language skills. 

14. WPA Board of Considtant Evaliiators. "Writing PrograuJ Kvalnation: 
An Outline for Self-Study. " Journal of the Council of Writing Profiram 
Evaluatofs. 4 (Winter 1980): ^3^28. 

15. Don K. liardner. "Five Kvaluati(m Frameworks: Implications for Deci- 
sion Making in Higher Kducation." Journal of Higher Education 48 
(1977): 574. and House. Etaluatina with Validity, p. 36. 

16. The literature in this area is large, hut see. for example, k. R. Ham- 
mond. C. j. Hursh. and K j. Todd, "Analyzing the Components of 
Clinical Inference." Psychohmical Review 12 {l^y): 215-24; P Slovic. 
"Analy/ing the Kxpert Judge: A Descripti\ e Study of a StcK-khroker's 
Derision Pnxesses." Journal of Experimental Psycholofiy 78 (19(>8). 
nmnograph supplement, no. 3. pt. 2; and P Slovic and S, C. Lichtcn- 
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» slciii, "Comparison ol* Buvesian and Krur(*ssion Approarlics to the* 
Study of Iniormation PnKcssinjj in Judgment/' Or^auizatioual Hvlun - 
ior and Htwmn Perfonnautv 6 (1971): 649- 744. 

2. The Quantitative Model of Writing Profiram Evaluation 

1. The Effectiveness of CoHe\ie4A'vel Instruction in Freshman Composi- 
tion, Final Report. Project no. 21H8 (Washington. IX): Office of Kduea- 
tioa. U.S. Department of Health, Kdueation, and Welfare. 1969), A re- 
|X)rt devoted to the pilot phase of this stud> was issnetl hy the same 
authors; see Interim Report: The Effectiveness of Collefie-l^'Vel In- 
stmction in Freshman Composition. Cooperative Keseareh Project 2188 
(Odar Falls: State C'ollcge of Iowa, 1966), A follow-up stud\ using the 
data collected hy Jewell. Cowley, and Khum, plus additional data col- 
lected from Uniwrsity of Iowa students invoU cd in the l*M studx; was 
(completed hy Richard Bradd(K*k, See his Evahation of Collene-lA'iel 
Instruction in Freshman Composition: Part II, (\>operati\e Research 
Project no. S.26<) (Iowa City: Tniv. of Iowa. 19{)8), 

2. For an explication ot those ETS procedures, see Fred (>odshalk. Frances 
Swineford. and William E. CioHman. The Measurement of Writing Abil- 
ity (New York: College Entrance Examination Board. 1966), 

3. Donald Wesling. John Conlisk. Sharon Evans. \\\ (;, Hardison. Ralph 
lioveherg. Emory Iblherg. and Jane Watkins, Evahation of the Four 
Colleffe Writing Programs at VC San Die^o (San Diego: Univ, of Cali- 
fornia, 1978). 

4. Molfetts statement. Evaluation of the Writing Programs at the Uni- 
versity of California San Diego." appears as Appendix II in tlie Wesling 
et al. report. Wesling and his colleagues take issue with a numher of 
Moffett s recommendations and criticisms. For an outline of these is- 
sues, see Wesling et at. pp. 12-18. 

5. On the matter of variahles left uncontrolled during the (*valuati(m. see 
Wesling etal., pp. 34-35. 

6. See. for example. Codshalk. Swinefbrd. and ('ofiuian. The Measure- 
ment of Writing Ahilittj and (Jharles R. Coopen "Holistic Evaluation of 
Writing/' in Evaluatinn Writinn: Measurinu, Describing. J udninn. vd, 
C;harles R. Cooper and Lee Odell (IVhana; National Council of Teach- 
ers of English, 1977). pp. 3-31, which provides good hihliographies of 
works dealing with the evaluation of student writing. 

7. Donald Daiker, Andrew Kerek, and Max Morenherg, "Sentence Com- 
hiuingand Syntat^tie Maturity in Freshman English." Colleue Composi- 
tion and Communication 29 (1978): 36-41; Morenherg. Daiker, and 
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Keri»k. '*Si»nU*ur(' ('omhiniim at \\\v i]a\\i%v An Kxpcriincntal 

Sliicly/* Hvscairh in the Tcarhinu of EuuHsh 12 (IfiTSh 245-.i(i; and 
Kcri»k. Daikcr, and Mon^nhcTU. "Sonlcnir OonihininU and ('ollcuc 
Composition/' Perceptual a ml Motor Skills 51 (1980): 1059- 1167 (Mono- 
graph Snpplcnu'nt 1-V51). Scvrral (jnrstions wvrv raisrd about 
slndy as it was reported in I97S in College Compositiou and (Umimnui- 
av ion, in Hcscarch in the Teaching of Entilish. and at the MVoni'iuu 
(.onliTcncr on Frtshniaii and Sophonion* Kn^h.sh \svi\ lor i»xanipU». 
Kinnravy, "Srntrmv C'oiuhininu in •» C!oniprrhiMisi\ c UuiKuanc Kranii*- 
work"; MfHon, "Issues in ihi* Theory and Praitire of Sentenir C'oni- 
biniii^'*; iipil Witte. "Review of Sentence Comhininu (iml the Teaching 
ofWritinu" Collvue Composition anil Communication 31 (1980): 43^}- 
.37). Because Sentence Comhininu and College Composition addresses 
several of'those (piestions. we assume that it represents the Knal report 
on the Miaiuit^tudy The loUowin^ r(*\ iew is hased on tlieir most reeent 
pnhiislied report 

8. *lrue etVntrol uroups — students ottered no instruction in wHtiui* — are* 
often ditficult to ohtain.i Many institutions arc* eomuntte<l to plaein^ all 
ireshuieii in eompositiou courses duruiu th(*ir first semester in eolleue. 
For example. Leste»r Faiuley was denied a retpiest to form a control 
^roup hy askinu sehrted stu(h*nts to postpone freshmau Knulish for one 
semester. (Sec* "The Influence of (Generative Klu*toricon the Syntactic 
Fluency and Writing Fttectiveness of (lolle^e Kre .lunen, " liesearch in 
the Teachinu of Eiialish 13 (1979); 197- 2(H).) 

9. Sentence Comhininu and Colleue Composition, p. 1101, The expiTi* 
nu'ntal uroup used W illiam Strongs, Sentence Comhininu. A Compoy 
inu Book (New York; Kandom. 1973) as their t)nly textbook. 

10. The curriculnin of the control sections, in fact. foUowed the oruaui/a* 
tion of James Mcfirinnnons Writinu i^'ith a Purpose, (ith ed. (Boston: 
Houuhtou, 1976), A reader was also used. 

11, Syntactic finency refers to the relative sophistication with which writ- 
ers of ditierent am^s are able to reduce clausal structures to less than 
clause status and to embed those rediu'tions in the sentences they write. 
Tin* t(»rni. syntactic fiuency, is frecincutly used intercbanUeabU with 
two other terms, syntactic annplexity Aud syntactic maturity. The phe- 
nomenon presumably referred to in these terms is frccjuently uiea- 

_ surcd by siich indices as uu*au t-uuit Icnuth (a t-unit beinu an indepen- 
dent clause plus all of its subordinate elements), mean clause* h*nuth. 
and mean nund)er of clauses per t-unit. These nu'asures derive Irom 
the work of KeHouu Iltnit. nmch ol which is summari/ed in his "Karty 
Bh)oiuinu and Late Blooming Syntactic Structures/' in Itlvuluatinu 
Writinu^ Descrihinu. Mca.su tin u. J uduiuu- W- 91 --KM. Lester Huuley, 
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in "Names in S(Mrrli of a (loiitt^pl: MaUirily, FliiiMu y. (loiiiplrxitx. and 
(irowlli in WrilttMi Syntax," doUctlc ('Omposition and (Uymmuniratum 
31 (198()V. 291-3(M), armu»s llial for stntlnits l)ryon(l hiuli sriiool. it is 
impossible to trll what aspects of lanuuaue (levelopini»nt these terms 
r(»f'er to. 

12. Moreni)eru, l)aiki*r. and Kerek, "Sentenee (.'oini)ininn at the (lollem* 
Level/* pp. 2«'Vi--55. 

13. Kllen \V. Nokl and Sarah W. Kreedman, Wn AnaK sis of Headers' Re- 
sponses to Kssa>s/' Hrmnrh in tlw Tcachinti ofFMiihsh 11 (UJ77): 1(>I- 
74; Faiuley, The InHuenee of (ii*nerative Hhetorie ' 

14. See, fi)r ivxauiple, Mi'Hon. "Issues in the Theory and Praetiee of Sc»n- 
tenee (^ond)ininu. ' pp. 2f)-34: KinncMVy. "Senteneevdomhininu in a 
(!onipreht»nsive Ixinunaue Framework. pp. (>6-67; and Witte. *Ke- 
vi(»\v of SeHferire Comhinmi^ ami tlw Tcachiui^ofWritinU- ' 

15. Truusfonuational SnitnuT Comhiuin^: A Mrthul for Enluniriuii Stpt- 
tactic Fhicm if in Kn^/i.\7i (Unnpositiint. Heseareli Keport no. 10 i(^ham- 
pai^n: National Oouneil of 'leaehers of Knulish, IWW). 

Ifi. Sentence Comhinint^: Improvimi Student Writiu^^ Without Fonmd 
iwtwnmur Instruction, Kes(*areh K(*port no. 15 (dhainpaiun. National 
('onneil of Teac hers of'Knulish. 1973). 

17. Daiker. Kerek. and Morenheru. "Usinn 'Open' Sentenee-C.'onihininn 
Fxereises in the (iollej^e Composition (.'lassroom. in Sentence C.'om- 
Innintiiind the ieachiufiofWritinf:, p. 108, 

18. Many of the rhetorieal prineiph*s were ohvionsly drawn from the work 
of Franeis CJhristensen. as e\ideneed in Stnniu's Sentence dtnnhinini:: 
A (Unnposinu Hook. The North Dakota stndy of jj;enerative rhetoriv - 
which was desi^ned. in part, as a replication of the Miami study — 
yiehled similar resnlts in hotli syntactic features and holistic scores 
(Faiul(»y» The Influence of (Generative Klietoric**). Althouuh the North 
Dakota study has many of the same weaknesses as the Miann study, the 
rhetorical assumptions w(»re unule more deliberately a part of a curricu- 
lum, primarily because the (!hristensen niat(*rials make many of these 
rhetorical asstnnptions explicit. 

19. On these* niatt(»rs. s(»e tluMuvestinators' comparisons of the two courses, 
pp. WJ()-11()3. 

20 Stt^phen F. Witte antl LesttT Faiuley. A Comparisim of Amdijtic and 
Synthetic Approaches to the Teaching of ('.(dh'nv Writing, T\\'H(G He* 
siMreh H(»pi)rt no 1 (.Austin: D(»partment of Kimh'sh. l'iit\ . of Texas, 
1981). KHIC D(K, no. FD209H77. 

21. A Theory of Disctmrse (1971; reprint. New York: Norton. 1980). 

22. Th(»se piirposes and modes of discourse* were pr(»sented through a course 
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sylluhns suppltMiicutftl. wUvu a\ailal)lt». hv parallel trtMtnUMits from 
Mk hat'l K. Adi-lstrin and jtMu Pix al s Tlw Writina Commitment (Ni-w 
York: Harcourt. 1976) and l)y excMiiplarv r(M(liims from Kamlall K. 
Dfckt-rs M.I Patterns in Exposition 6 (Boston: Littlr. 19781. For in- 
struction in tlif convtMitions of standard written Knulish. students re- 
lied on Jim W CordtTs Handbook of Currnit English (CAv\\\iv\\\ IL: 
Seott. Foresnian. 1978). 

"\ Coneeptual Kln»torie of tlie Composition." Collruc Composition and 
Communication 22 <1971); 348- 54. 

Susan \\* Wittiu. Stvp\toStntvture\C/d\u\}rii\u,i\ M\ Winthrop. 1975). 
Because the text materials for the meristie option relied on consider- 
able Krammatieal terniinolouy. teachers employing this approach used 
J. C;. Blumentliars English :l2(Hh A Programmed Course in Crauvnar 
andCsafie (New York: Harrourt. 1972). a proKrannued urannnar text to 
tiMch students grammatical v(Hahnlary and to teach them conventional 
usaue. As part of the second opti<n>'s currienlum. English :i2(H) was in- 
tended to provide students with the competencies that wouhl enable 
them tocomph»te the more sophisticated M»ntcnc(»-h»vel exercises avail- 
able interacti\ely by computer. See Susan W. U'ittiu. Dialouue (Iowa 
City: Conduit. 1978). and "Diahiuue: Project C-BK Drill and Practice," 
Pt;;<7oie4(1978); 20-22. Thes(» exercises, in turn. prepare<l students to 
handle the treatments of paragraphs and essays in Witti^'s Steps to 
Structure, 

The r(»search desiun as wvll as the comparisons are present<'d in detail 
in Witte and Fai^lcy. A Comparison of Analijtic and Stpithetic Ap^ 
proaches. p|). 7-10. 295":}(K). 

}oh]\ A. Daly and Michael D. MilltT. "rhe Kmpirical Dc\elopm(Mit oi 
an Instrument to Measure Writing Apprehension." Heseareh in the 
Teaching of Enulish 9 (1975): 242-49, 

A. L. HayKor. MeCrau-llill Hasie Skills System Heading Test: Exam- 
ine r's Manual [Sew \m\i: MtCrawHill. 1970). 

A 1. Hayuor.. Stearau-llill Hasir Skills Stjstem Writinu Test Exam- 
iner's Manual {\e\s York. McCrau-llill. 1970). For the two MeCiraw- 
Hill tests. Forms A and B wer(» used, with one class on each side of each 
principal comparison receivinu Form A as a pretest and one class on 
each side of each principal comparison receiving. Form B. For the post- 
test the forms wer(» reversed for each section. 

The topit s are rt^producetl in \\ itte and Fai^lex. A Comparison of Ana- 
lytic and Synthetic Approaches, pp. 13-^ 16. Posttest data were coika ted 
duriuK the last \soo\i of class(»s and durinu th(^ final (Examination period. 
Because* the essays tin stiuh^nts w rote were used in couiputinu c<mrse 
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Krades, wvrc photcK-opied tor holistit; storing prior U) Uieir having 
bcvn marked by llie teaelievs tor n^turn lo i\w shulenls. Stiidenls were 
given exira eredH for eoinpleling ihe Uvo MKiraw-Hill lesls. 

31. Interrater reliabilities tor ihr Iwo narralive-deseriptive <»ssays were 
eoinpuU»d lo be .76 and .78, ri»spec1iv(»ly; and lor ibe Iwo argumenta- 
tive essays, they were eoinpnled lo be .82 and 81. respeetively. Tbe 
MrGrnU'Uill Readinfi, Test and UVifiim /i'vf anil ibe Miller-Daly WAT 
were eonipuler stored. Oala trom eatb eoinparison were snbmiHed to 
analvses of t*«)variant'e, wilb pretest stores used as tovariates to control 
lor pretest ditlerenees. 

32. See NVitte and Kaiglex. A Comfmhson of Analtjtic and Stjntlwttv Ap- 
prtHuiws, pp. 127-242. 

:J3 Tlies(» analyses are presented in U'itte and Faigley. A Comparison of 
Analtjtic and Stpitlwtic Approaches, pp 243- 49. See also Faigley. Daly, 
and NVitte, "The Hole ol Writing Apprebensi(m in Writing l\»rtbrniante 
and Coiiipetente." Journal of Educational Research 75 (Sept.-Ott. 
1981): 16-21. 

.34. Type 2 (»rrors (Hcnr when statistical power is inade<iuate to attept tbe 
null hypothesis. Statistital power in the 'lexas stmly was low owing to 
the small ntimber ot snbjetts in each tall. B. j. Winer suggests that .30 
and .20 levels ot signifitante may be more appropriate under these t<Hi- 
ditions than .05 and .01. See Statistical Principles in Experimental l)e- 
sitin, 2d ed. (New York; MtCiraw-lIill. 1971). p. 14. 

:i5. On this point see Lee Odell's distussicm in 'Defining and Assessing 
(:«)inpetente in Writing." in The Sature and Measurement of Compe- 
tency in Kfifi/is7i. ed. Oharles H. Coopj^r (Trbana: National tyouncil ol 
Teachers ot Knglisb. 1981). pp. 107-8; Gabriel Della-hana. Lee Odell. 
Charles (:(M)per. and Cieorge Kndo. The Writing Skills Decline: So 
What? ' in 77i<' Test Score Pecline: Meanimi and Issues, ed. l^wrente 
Lipsit/ iKnglewocKl Clitts: Kdutalional lechnology hiblitati<ms. 1977), 
pp. 163-86. 

.36. j. C. Seegiirs. The Form ol Discourse and Sentente Structure." Ele- 
mentartj English 10(1933); 51-54; Lois \. Johnson, "Children's Writing 
in Three Forms otComposititm." Elementanj English 44 (Mil): 265- 
69; L. Hamon Veal and Murray Tillman. "Mode ot Distourse Variation 
in the Kvahiatiori ot Children's \yriting/' Research in the Teaching of 
Euf^lish 5 (1971); 37-45; Marion Crowhurst and (ieiie L. Fiche. 'Audi- 
cnce and Mode ol Disnmrse KHects on Syntactic Complexity in Writ- 
ing at Two Crade Levels. ** Research in the Teachina of English 13 (1979); 
101-9; Faigley. "Names in Search ol a (^mcept. 

37 As lar as we know, composition ri^searchiTs have always treated holistic 
scores as thoiigh they represented a cr)ntinnous variabh*. like age or 
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wciKhr H(wm»r. Iiolistu' (»vaUiati(ni in\ol\i's plariim rssavs intci vaiv- 
^oricvs or uroups aaordiim to ihvir rclahve (jualily. luUkv a variahir 
siicli asam\ a holishr scor<»can hr no liiuluT tliau ihv imtnlKT assiKiiod 
lo that particular cat(»uory of i»ssays. {\\v minuTiial "\ aluc" of the ivssay 
rcprcsnitiiiK simply a naiin* Jor a cat(»uory. It may hv aruucci that it 
makes no more svusv to treat a holistic score as tliou^h it w vrv i? imiuTi- 
ral value than it does to assiuii a nunibcT name to sex or race und then 
treat the data as nmnerical. W hat would it mean if \V(» were to sa\ tliat a 
Kiven |Mipulation had an average racial makeup of 4.85? Perhaps holistic 
scores shcndd he tn»ated as categorical rather than numerical data. This 
is, ho\v(>\ er. a topic far too technical for hdl tnMtnu»iit heri\ althonuli it 
is oni» to which resi»arch(Ts should address tlicmscUcs. The i>sue, of 
course, has a nnndxT of implications not (udy for the evaluaticm of writ- 
inu i-ourscs and prnnrams hut for writing research in ueneral and for the 
work of the Kducatioual lestin« Servii e in particular. 
38. Sarah \V. Freedman. "luHueuccs au Kvaluators of K%i)ository Kssays: 
Beyond the 'lext." Research in the Teuchina of Eufilish 15 (1981): 245- 
55. and Freedman and Hubert CJalfee. "Holistic Assessuieut of \Vritin«: 
Kxperimental Desiiun and Cognitive Theory" (unpuhlislied MS). 

3 . A Framework for Evaluatina Colhfic Writina Programs 

1. Kalph \V. Tyl(»r. Husiv Priudples of Curriculum and Instruction (Chi- 
cau(i: Vwis ofChicauo Fr. 1950); and "The Functions of Measurement 
in Iinpn>\ inu Instruction." in Educuiionul \teusurrment. ed. K. Ki,in-" 
quist (\Vashinut(m. DC: American Council au Kducation. 1951) pp. 
47-67. 

2. On this luMivr. see Arieh Lewy, llie Nature of Curriculum Kvalua- 
li(m." in llamlhook of Curriculum Eniluatiou. ed. Arieli lA^wy (Faris 
and New York: UNKSCOand I^jn^man. 1977). pp. 10-^11. Tyler's model 
has hveu critii i/eil hy KohiTt K. Staki». "Umuuaue. Kationality and As- 
sessment." in Impnn ina Educutiouul Assessment, ed. \V H. Heatty 
(Washinuton. DC: Aswxiation for Supervision and Curriculum Devel- 
opment. 1969). pp. 14--4(), for not ailc<|uately aironimodatinjii pnKvss 
varial)l(»s or the conditions which Kivi» rise to products. The Tyler model 
has also hei»n criticized for similar reas(ms hy Michael Scrivcn. "The 
MetliiHlolo«y of K\aluation. ' in Perspectiivs on Curriculum Evuhuh 
tion. ed. Kalph VV. T\ler (CliieaKo: Han(> McNalK; 1967). pp. 39-83: 
and by (;. V (;lass. The Growth of Evahtation Methodology (Hcndder: 
l^lMiratory of Kdncational Kesearch. Univ. of Cohmuk). 1%9). 

3. On tlus matter, see Hous(\ Eaduatina with Validity, p. 27 
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4. Sve StepluM) F. \\\lh\ Rciurr I). (rht»rr\. and Faiil H. Mcvvr. The lioals 
of Freshman U niin^^ PwnramM as Pvrcnied by a Sational Sample of 
CoUeiieaud Vnii'vrsitij WritinuProfiram Director^ ami Teachers, FIFSK 
Crant (;(N)8(M)5S9fn Toihiural Kr|)(irt no. 5 (Austin: Wrihim Fniuraiii 
AssessimMit (Mfiir. Univ. of Texas. 1982). KKIC Dm . mi. KI>21(r 395. 

5. See. for example. Jaek (i(«Kl\. DomvHficafum of the Sarofie Mvid (('am- 
bridtte: Cianibridue Tiiiv. Pr. 1977); Uferanj in Trattitiunal Societies 
(Cainhridue: (?anihridee I'niv. Fr, 1969): Krii A. Havehnk. Ononis 
of Western Literacy {Toronto: Ontario Institute for Studies in Kduta- 
tion. 1976). 

6. See Richard Ohniaiin. English in America (New York: Oxiord rniv. Fr. . 
1976); and Shirley Brice Heath, "The Functions and Uses of Literacy/* 
Journal of Communication, 30(1980); 123-33. 

7. Heath. "Toward an Klhnohislory of NN'ritinu in .American Kducation." 
in Writing: The \ature. Deietopment, ami Teachinu of Written dont' 
munication: Vohnne L Variation in Writing: Functional and Linuuisttr- 
Cultural Differences, ed. Mareia Karr Whitenian (Hillsdale. NJ: 
Liwrence Krlbauni. 1981). pp. 25-15. 

H. Lester Fai^ley. Thomas F Miller. Fanl K. Meyer and Stephen F N\ itte. 
Writinu after Colleue A Stratified Survey of the Writinu of Colleae- 
Trained People. FIFSK Crant C;(K)8(K)589f). Technical Keixirt no. 1 
(.Austin: VVVilinK Frov^ram .Assessment Ollice, Vniv. of Texas. 1981). 
KRICDiK-. no. KI)210 7t)H. 

9. T Neville Fostletliwaite. T)(*terniinati(ui of (General Kdncational Aims 
and Objectives/' in llandhooh of Curriculum tWaluation, esp. pp. 
56-6(). Fostlethwaite's Muirce is F. B. \\onier. What Is Sational Assess- 
ment}^ {\nu Arlnir: National Assessment of Kdncational Frouress. 197(1). 

10. In fact, the Oomptroller (ieneral's Ke|Mirt to the (ionuress, The .\«- 
ti^mal Assessment of Educational Prouress: Its Hesults Seed to Be Made 
More tSr/(i/ (NVaslun«ton. DC: U.S. General Account inu Office. 1976) 
criticized the first assessments for NAKF's failure to take* into amumt 
the students' varying Ixick^numds. 

11. See Hoy I), (^(ddnian and David J. Hudson. A Multivariate Analysis of 
Academic Abilities juid Straleuies for Successful and Uusuce(»ssfid Col- 
leue Students in Difierent Major Fields.' journal of Educational Psy- 
chology 65 (1973); 364-70. and Hoy 1). (;olduian aiul Kcbeaa Warren. 
"Discriminant Analysis of Study Strategies (loimected with (irade Suc- 
cess in Different Major Fields."* Journal of Educational Measurement 
10 (1973): 39-47. 

12. Carl Frederikscu. Abilities. Transfer and Inforiuatimi Retrieval in Ver- 
lial LtMrniutf. Multivariate Heharior Research Monographs 2 lUKi9): 
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1:1. H. Haviiit^liiirst Detvloimwutul Tasks ami Educatum, vA. (Now 
York: Diivsd N!i Kiiy. 1973). 

14. Then* IS ;i loiisiclrralile litrniliin* iivailaliU* on llu* siilijtTi oi aintcxtiiai 
iiilliuMicvs on cdtKational prouranis. Svv, for cxuinpks Kavniond S. 
Adams. Kirhard M. kinil)l(*. and Marjorit* Marliii. 'Scliooi Si/.v, Orua- 
iii/;ilional Strncttirr and TiMclnnu Practices/* Eilucatioual Atlmmist ra- 
tion {hmrtirhj ft (Auttinni 1970): 15-31; Kuon il iUih\ and Charles K. 
Hklwell. Administrative Helationships: Teaclwr Effvctivenvss, Tvaclwr 
Satisfaction and Administratirv Af/ir/t tririr (C^hkauo: Midwest Adniin- 
islralion Centre, Unix, of Chieauo. 1957); and Kalph \V. Ixirkin. "Con- 
textual laHnenees on Teacher t^*adership Styles." Saciolotitf of Edu- 
cation 46 (1973): 471-79. Jacqne Har/ini in his 196()s treatise on the 
American nniversity. (hscusses many political and social inlinences oi 
context on hiuher education in this country See his The American Vm* 
vrrsitif: How It Huns, Where It is iMnu (New York; liarfier, HKiS), 

15. "Divinu hv An Intrmlnction to Basic Writing." College Composition 
and (Communication 27 (19761: 234^-39; Errors and Expectations: A 
iluide for the Teacher of basic Writing (New York: Oxford Univ. Fr.. 
1977^. See also Sliauuhnessy s review of research in hasic writing: "Basic 
Writing/* in Teochinu Composition: 10 Hihlioiiraphical Essays, ed. 
(iary Tate (Fort Worth: Texas Christian Univ. Fr. 1976). pp. 137-67. 

16. Many writers miuht he cited here. .Anninu those are se\ eral of tlie es* 
sayists represented in two rt»eent collections: Walker (iihson (ed.) \e\c 
Students in Two-Year Colleiies: Ttveh e Essays (Urhana: National ('onn- 
eil ofTeaehersof Knulish. 1979) and l^iwrence N. Kasden and Daniel B. 
iioehrr (eds. ) Hasic Writinii: Essays for Teachers. Researchers, and Ad- 
ministrators (IVhana: National Ctnnicil of Teachers of Knulish. 1980). 
On some of the specific needs of hasic writers. Andrea A. Lunsford's 
"Cognitive Development and the Hasic Writer/' College Eufilish 41 
(Sept. 1979): 39-46. is an especially imiMirtant essay Also relevant is 
Muriel Harris's "Individualized Diapiosis: Teaching for (Causes. Not 
Symptoms, of Writinin Deficiencies/' CoUeus' English 40 (Nov. 1978): 
318-.'J.3. and Harvey S. Wieners "Basic Writinji: First DaysThouuhts 
on Process and Detail/* in Eiiiht Approaches to Teaching Composition, 
ed. Timothy B. Donovan and Ben W. Mi<Jlelland (Urhana: National 
Council of Teachers of Kn^lish. 1981). pp. 87- 99. 

17. Ix'onard Butnum. Planninii Useful Ecahations: Eialuahility Assess- 
ment, with a **Fonvar(r* hy Joseph S. Wholey (Berkeley and London: 
SaKc, 1989). esp. pp. 89-1(4. 

18. See Witte, Meyer. Miller, and Fai^ley. A Sational Suney of CoUene 
and University Writing Program Directors, pp. 104^5. 
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19. For ilisiiissioiis by Viirious hands of "wriliim jhtoss {\w c urriruliim" 
miuviiu iii. sfcTohy hilwilcr and Art YoiumirdsJ. iMtniuatii i onnvC' 
fiuns: Wntina oud Hvudinu Aiross the Cutrimlum iTrliana; National 
Conniil oni ailtcrs ol Knulisli. mil Sec also iVtc rs. "WritinK Across 
th(*(!iirrii'idinn.** 

20. A lair anioniit ol' tins ri M arili is siiniiiiari/rd in Kri( Brrdo. "Con- 
trxtnal Iniltii na s on Tratlirrs' Iii.li-ui tinnal Approarhrs. " Jounud of 
Cinrirulum Stutlivs 12 (WHO): 

21. Svv U ittc. Mrvrr. Millc*r. and Kiiulcy. A SatUmul Sun rtj of Collvu^* 
and Vninrsitij W ntitta Program Dhvvtors, esp. pp. 112- Hi 

22. (iillwrt Hylr. Tlw Comvpt of Mind iSvw York: Harnrs. IW)). pp. 

2.1 (lonsidt raliic i*vidc*iiivol hotli kinds oFi iirriiiilar variables is presented 
in Stephen F. W'itte and Faiil H. Meyer with 'I'hoiiias I' Miller. A Sa- 
tiotml Survvtj of ColU iiv und luiversity TvucIuth of Writing. VIVSE 
(;raj«t iAn\m)nHm, leelmieal He|>ort nc*. 1 (Austin: Writing Frourani 
Assessment Ollice. Univ. ol'Texas. 19H2) KlUC Doe no. Kl) 219 779. 

21. S(»e Hiehard K. Yininn. "Faradimns and Prolileiiis: Needed Heseareli in 
Klirtorieal Invention." in Hcseurch on Composing: l\nnts of Ovpav- 
turv. pp. 29^47. janies A. HeHin and Hohert V Inkster. 'Current- 
Traditional Khetorie: Furudiuni and Fraetiee." hrshman Enalish Srws 
no. 3 (Winter I9H(M: 1-1. 13-11. Donald C. Stewart. 'ConiiMisition 
TexthiNiks and tlie Assault on Tradition.*' Collvac Comiioxition and 
Communiration 29(1978): l7U7(i and Maxine llaiiston. The Winds ol 
(ihanine: Thomas Knhn and Kevolntion in the Teaehinn of Writinu." 
Colleuv Composition and Communiration 33 (I9fi2); 7(i-8H. We iiiiulit 
speculate that the traditional paradimn is not likely to change (*ither 
snhstantially or quickly unless and luitil composition teachers acc(»rd 
textlMM)ks considerably less status than they presently lia\e in the teach- 
iiiUoiWritinu. TcxtlnMiks, to be sure, are usehil lor teachiim content in 
tin* sense ol* what, but they are of considerably less value in teacliinu 
content in the sense ol /loic. The reason lor this, ol course, is that they 
can oidy address how by converting it to what. On this latter issue, see 
also Mike Kosc. ' Sophisticated. Inellcctive Hooks— The Dismaiitliim 
ol FriKrss in C.*oiii|Misitiou Texts." CoUvfi*- Composition and Communi- 
cation 32(19H1): (k>-7I. 

25. W'itte. Me yer. Miller, and Kiiulex. A Sutional Suncti ofColhui' and 
Cnii vrsity Wfitinu Vroitram Dirvctnrs, pp. 57-1^). 

2^). "Composition TextbiMiks and FedaKoi;ical Thcorv. A He\iew-Kssay." 
VolhiLv Enifjish 43 (19HI): :}93-4(W. 

27. Woods acknuwiedues Ins debt to Harry Kinll. "l)c\elopinental Ferspcc- 
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h\c'S .mil tin* liMt'liiimnKlomposihoii/' ( W/c^jr Eualish 41 il9S(H; 711- 
52, liir tliis ivnu. 

2H. Stfplirii Jiiilv's Till* KxpiTiiunilal Appro.uli: Iiuirr Worlds lo Ouli r 
WorliK. 'in F.iuht Apprtmchcs to lhi('hiuu(-(nt^^^^ vd. rimolliy H. 
Donovan ami Hrn \\ MiClrllancI irrliana: Nahonal Coinu il oriracli- 
vrs oi Knulish. pp. ;17-5I. proviilrs wlul wr llnnk is a uimmI ai- 

onnil ol lln' rxprrssionist" approaiii WimuIs drsiTiiirs. 

2!J. "I'hr Sfarili lor Inlrlliuililr Sirurhirr in ihr Trai liinu ol ( 'oniposilion." 
( t)lU%e Criiii/iovirinn tml ComumuHnUon 27 (Ifj7(i): 1 12- 17. A sonn»- 
whal Ifssanihihons. Iiut n«niftlu»lfss valiial)li\ (iciinrahon ol an nnilrr- 
Ivinu strni'tnrr lor Irailiinu loniposihon is loiinil in Oaroliur I). Kik- 
lianll and Donald C. Slrwarl. "loward a riuulional Ta\ononiy of 
CoMiiMisihon." Colleuv Comitosition ami CommuHuvtiou 'M\ (U)79i; 
.'VJH- 12. Ki'kitardt and Slrwarl distinKuish hrlwrni hvo approadu*s to 
f4nn|)i)sUioii. one llironuh tvrhuufues and n\w Oirouttli imrpnses. Tliis 
is rssrnliaiiv adistindion hrlwvrn /lofr and nhf, l)rKviM*n ffirz/nv and 
ends, hviw vru ifir'r/rs and aims, Tiiis dishnrtion is iMiiit into D'Aiiutdi/s 
structure. hiif^Kckliardt and Strwart strrss tlu» priniai v o| pnr/invi'vand 
oHi»r a taxonomy ol pnrjjosrs wliu li dillcrs in important ways Irom tlir 
oni* Imilt into D Auj^tdo s strnrturr 

K). Fualish ('(nuinmtum ami lih tonr, rv\ AuuTiian rd {\iw York: D. 
Appli'tmi. IHm 

A Thrnrif of Dimmne For Irss thoronult f vatnuMils. sro his 'Hasit 
Aims ol Disconrsi'." (!nllm'( omitositum aiulCommuuu atiou 2(MUKW^: 
2^)7- iI3. and liis roiialiorativt* work with Jolm (aipe and J. \V 
Campln*!!. Aims ami Amlinurs {lh\\m\\\v. KtMidall/Hnnt. W7(il. Kin* 
nfa\y*s throry ol'disronrsr nnnlfs is set down in anotluT itdlalMirativi* 
work with ('opi' an<l Cainphrll. Writiua — Hasir Modes of Oruamzatitm 
(Dnlnnpuv KiMnlalWIlnnt. IM7ftK Hjihard II. Ilaswrll. "Ilutiis of Dis- 
course; A (iiassiliiation lor Stmirnt Writers. ' ('olleav Knfilish 4.1 {Mt. 
HiHlh Ib8-7S. raises a nnnduT o| ohjeetions on peda^ouieai Uronnds io 
Kinmsuy's theory of aims and modes. Lee Odell. Teaeliers ol (aim- 
IKisitimi inn! Needeil Kesi*ardi in Discourse Tln»ory." ipiestions the 
iiasis ol Kinneavy's ehiim tiiat aim or purpose is ail important. 

32. See Tntrhina the I 'mmsv of Disroarse (Boston; Hoimhton, IWiH), "A 
Stnutural Curritulmn in Knulish." Harvard Edmatiomil HvviiU 3ft 
rlW)l><: I7-2H. and "A Katiomde tiir a New (liirrieuluni in Knulish." in 
lihetorii: Thvtnivs for Application, ed. Hohert \|. (*orrell (Cliaiiipaimi; 
National Cloimeil ol TeaehiTs ol Knulish. IW7». pp. 111-21 

33. See Hritton, HiirUi^ss. Martin. MeKeod. and Hosen. Thv IMrlopmntt 
of Writing Ahilitii s i/'-M' iKonih>ii Maeminaii Kdueation Kid. Ui75). 
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and Hritton. "TIk* C!(iiii|XiNiiiu IViktsson uiul the* Kuiiiliniis iil Writini;/* 
ill lU'svarch on Composmn: hnuts of l)vi}artuvr..\^\t. I.'1-2H. 
'M. Svv]. A. Villi Kk Willi uiiilriliiili(Mis by L. (i. Ali'Miiifler. Thr Thrvshold 
h'xrl for Modvrn hnitiuailt* iM^arninu in Schools d^nidon: l.(iiminiiii. 

Vi. 'Tlu'orirs iii Ooiiipnsilinn and Actual \Nriliiiu/' Kamas lUiulish 59 
(l)rc. 1973): 3- 17; Inr a iiitirr rt'UMil slatciiUMil. In^l•llll•l willi a uriMtcr 
liiHtnrical (r<Mlnu'iit, svv Kinii('a\ \\ "I'raiislalinu Tlu*orv inin l'ra< li(r 
in T(*acliiim Compnsilinn: A ilislnrkal and a (innlrniporarv \i[ir\v/' in 
Classical Hhvtohc in Modvru Disvoursv, i»d. HnlirrI J. (Inniinrs. Lisa 
Kdr, and Andrra Lnnstord ((.arlxnulair: Siinlhrni Illinois l^iiiv. lY. in 
prrssl 

•Ifi. I'lir hvsi nKKlrrn rrprrsnilalivr ol this approach is Kdward V. ]. (.'or- 
Im'II. Classinil Hhvtorif for tlw Modem Stutlmt iSvw York: Oxford 
l iiiv. lY. 19711. and "A Nrw hxikat Old Hlu lorit /' in Hhvtoru : Tlwo- 
rivs for Applitvtiou, pp. Mi-22. Srr also John II. Macklin. Classical 
Hhctoricfor Modern Discourse: An Art of Im eutiou . ArraiiUt'^ueut, and 
SUjIv fttr Headers. Speakers, and Writers (\v\\ York: Vrvv Fr. . I9(i9^ 

37. Auordini; In kinnravv. llu* Uvsi iniKlcrn (*x(Miiplar ol this approacli is 
Francis (.'lirislciiscn. The Christensen Hhetoric Vrouram <Ncy/Ytirk: 
Ilurpcr. m>H) 9 

iH. Sec Ken Macroric. 7r//i»(! Wn'rinmKm'licllc Park. NJ: llaydcn. I97()> 
and Writina to He /{<7ir/(KtK'licllc Rirk. NJ: Havdcn. miHl 

39. Sec ( ;arl II Klans. Style in EntiUsh Frose {SvSs York Macniillan. 19()9). 
and (jerald Levin (ed.l. l*rose Models iNew York: llar(*<uirt. I973K 

•10. See Koherl Z4M*llner. Talk-NN'rile: \ Hcluiviortil FcdaKn(;\ lor (.*(nnposi- 
tioii." College Enalish :M (19691: 2fi7-320. For part ol the debate which 
ensued Innn Z<Hdlner's essa\. see '*On Zoellnifrisin. " College Eiiuli'ih 30 
(1969): Ci45-()8, which contains a nuinhcr ol rpixinses hy various hands 
to Zoellncr's hchavioral approach. 

41. See Peter Kllxiw. W Wfinu Without Teachers il^nido^ (hlord Tiiiv. Pr. , 
1973). and Unfina with Power. Techniques for Masterinu the Writing 
Process (Oxlbrd: Oxtlird Tniv. Pr.. 19H1) and "A Metliml lor Teachinu 
WritiuK.'* Collcue Enulish 30 tl9(>S). 115-25. U'illiani J. Coles, jr.. and 
\m disciples should also he included within this class. See (!oh*s. 'The 
leachiiiK "t NVritiiiK as Writinj;." College Enalish 29 (19fi7i: 111-16. and 
The Plural I: The Teachinu ofWritinu (New York: Holt. 197H). as well as 
Kenneth Dowst, *The Kpisteniic Approach: Writing. Knowinu. and 
U*ariiinK» in Eiuht Approaches to Teaching Composition, pp. 37-31. 
Also u champion ol'the "lA*ariiinud)y Doinu" inethtKl is James Moiiett. 
Teaching the Vniierse of Discourse, esp. chap. 6, pp. 1H8-2I0. 

42. See Sotes Toward a Sew Rhetoric: Six Essays for Teachers (New York: 
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HiiquT. I?>^>H^ Sniiir (if tlir ciirm nl.ir iitui iiistrmlUHKil vahalilrs in- 
iim4MirnihV(* rlu tork iin^disinsMul itt Nnilnii Kiiiuhnrii, Ia sIvt 
Faiulrv. aiul rhmnas CUmiumis. A SyntaiiirApiiroarh to Cullcai' Wht- 
mu An Analysis of Thvonjnnd Efftrt ((;raml Forks: I ni\ . nt Ntirtli Da- 
kola Shulv (•rniipnn K%alnahuii. MhV. 

iV Kir iliiistralinn. sw Fram is ChristiMisni, Tlw Chrishusrn Hht Umc 
PrauravL Htmmc}VdU ChrisUMisni. Tlw Clihstvmni Mvlhnd {Svw 
York; Harprr. 1979 Frauds ami Hoiiiiii'jraii ChrisU'iistMi. ASvwlihvUh 
he iSvw Vurk HarpiT. I97<>i. and Siisun WiUin, Su-ps to Struvturc. 

II Hn«rr VV. Slinv. "A Hulislii \ ifw o| l^nmiauf." Hrsvarcli in the Teavlh 
inu oj Eualish 15 (I9SP; 101-11 

15. Fur t'Hainplr. AiidrcM laiiisliird. "Ccimuliw Drwioptiirtit and the* Basic 
Writrr 

Hriia' II ]u\iv and M Uoil. SUidrls of Tau liinu (KnulewiMMl Clills. 
FrcMihiv-Hall. 1979). Scu' alsu joyic-V snnnnar\ slaUMnnil. "A Frohlrni 
ol Calcuorifs: Ciassilyinu Appnwdirs in UMihinu." Journal of Etlura- 
tion IW) ( AujU. 1978); 07-95. }u\\v ninfnids that two prolilrnis* onr lin- 
miistir and rnnirptnai alli'il disi iissions nl ti^arliinn iid\rrs«d\. Tlu» 
Krst prnlilcni is that tlii' trrin tcarhinu ^ nitni iisrd in an nndilU-rciiti- 
atc'd way. and thi» scrund is tliat schulari^hip iin trachinK biiscnl on 
"valnr nrirntatinns wliiili inasqneradr in analytic dress/' 

17. S(H\ tor rwinplc. tlir essays nillfdiui in V 1,. tVterson and II. j. 
VValhrru u»ds. t. Hi'svan hun 7rra7niit; (Herkdey. \l4<;ntdian. 1979), 

IS Fli/aheth Mi Fhersini. "Cuniposition/' in Tlw liutrhinuof tlndish, the 
Tfith Yvarhook of tlw Satumal Sodefyfor thv Study ofEduration. pt. 1. 
i d. Janus l\ ScpiiriMCiiiraUn; National Smictx lor tin- Stiidv »! Kdnca- 
tion. 1977). pp. 178^SS. 

19. On this matter, see John Dixon, (irou th in Eualish (New York: (hiiird 
Tniv Fr, 1W7). 

50 Si e Donald M. Mnrra\. A Writi r Tvachvs Writing A Pract\nd Method 
of Tvarhinti Composition (Boston: llon^hton. 19(>H). riionias Carniivlli. 
The VVritiim Cotderem e A One-to-One Conversation/ in Eiaht Ap- 
proavhvs to Teaching Composition, pp. lOl- Jl See also Charh s H. 
Dnke. ' The Student Centered Conterevee and tlie Writiim Frmess/* 
Enulish Journal f>t (Dei*. 1975): 1 J -.17. 

5F For a definition of the uinfereiuinn niethml, a disenssiim of the as- 
sumptions midtTlvinu the appriiaeh. and a ihstussion of the tcMehers 
role, see Carnieelli. " The Writiim Ctmfereiue: A Oiu'-tci-One Conver- 
sation/' pp. 1(12 19 Althcmuli Uieonifereiiiinu metluKl lufs received a 
Ureat deal of attenticm. knowh»due of how and wliy it works remains 
laruelx aneidotal in naturi* As far as we know, oidy on** n'searelier-— 
Sarah \\ Freedinan— is jm sently trying to exttMul knuwled^e beyond 
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tlu* It'Vt'l o{ iUv aiuHildlal Set* Uvr "K\aluatinn in llu* U'rihim Cdii- 
IrnMur: An InU-railiN t» PriKrss.' in SvlcrU^d Paptrs from the NSl lixas 
Writinu Hi'srardx ('onfrn'uvr. pp. 

52. Hiiurr (•arnsiin. 'ihw to One*: iutnrial ln>trurh<ui in rn'>liinan (jmu- 
|Hisitinti.' in \n< Pirn turns for Communitif ( (fllcars San Franc ism. 
Jossvy-Hass. 1^)7 1^ S V 

r%V St'v KiMinrlli Hrnllri*. TIh* liriHikKu Plan: Attainniu Inlrlln hial 
(iniNVtli thnnmli IViT-(»ronp Tntoiinu." Liln ral Edunilum (>l tl^TS'; 
n7^f>S. ami his "(.'fillaluirativr l.iMrninU; Scnnr Pr.uliial Mmlrls. 
ColUllv Entilish W (U)7^. KVI- I.V Hir ulhrr liratini'nts tii idllaltdia- 
liv(* Iraruinu and iUv usr iil prrr uninps. src Tlinui Hawkins, i^rtnip' 
luquinj Hrhniqurs for iW/r/imt' UVifin^ (Trliana: Nationiil ('tinniil nl 
'I'tMcluTs III Knulish and thr KltK ' ( !lrarinuli(MiM> un Htsulinu ancl ( 'uni- 
nnmitatinn Skills. Ui7(i). and "hitiinary and AnditMur I'ltr lU latitin- 
ship H(*tsvirn Ui'v isinn and tlu' Stnial Dinu'nsions (il IW\ '|ntt)rinu. " 
Collrae Entilif^h 12 (IUS(»: UH, and lUthanl Ct-lihanll. "Iraniuurk 
and rrrdhark: Hrnadc^ninu the Hast* oK 'iillalMiraliM* Writinu. " VolU'il*' 
Entilish 12 (WHO): (>^)-7i. Alsu rdrvant arr Ni il i:llnian, "W vr KNalua- 
tiiMi and Fi*i*r (iradinu, " English Journal <v) (Mar. 1975^. 79 HO. Allan A 
(Uuttluirn. "(^MipiTatr and (!rl^lt(^ TiMiliiim Writinu 'I'liniuult Small 
(FHinpsl" Enuilish JournnI h2 iDrt* W7.'i*. 1274 "-7.*). and rluMKlnrr \\\ 
llippli'. "Thr (iradiTs Hrlprrs — ( !(illrauui*s. h^rrs. and Snirccards." 
Eniilish lournal (A (Mas W72»: m)~m. 

.>!. Wittr. Mi'Vrr, and \lilli»r. in A S'ationnI Sun vxf ofCollrut* tnul l Uivrr- 
Hittj TeaclurH of Writtnn, rrport that tin* wsv ol prrr ttitiihnu. ron- 
trrrnnim» and rollahtirativr Irarninu vvi*rr tlu* inslntilional nirtluMls 
most oltrn nauunl as f/ir most \tu'n*ssful a^pirts nl thr tiMrhinu til a 
national sainpir ot thr best rollrur and nni\rrsilv writinu trat hrrs in 
thi» roiiiitry. 

55. Thrre is availahir a ImkK' oI litrraturr i>n thr tihsrrs.ititin and drsc rip- 
tion ol instrurtion. For rxaniplr. srr Ariio Hrllak. /Vtr iMnUOthlc of the 
('lassrottfu (Nrw York: Traihrrs ( jdlrur Pr.. HH>6). (iary I). Horirh. I). 
Malit/. («. 1>. Kuulr. and Ni. Fasriiiir. "(iouvtMUi'iit and Disiriiniuant 
X'alidity oi Fivr (.lassriNMu OhsrrNation Systrnis: Irstinu and Nhuh'l- 
inu. jourmxl of Educational Psychology 70 [ IHlH). 1 19-27, W \S\ ( a>olry 
and (i. Lrinhardt. "Thr Instrudiomil Dinirnsions Stndy." Educational 
Evaluation and Policy Analysis aim)) 7-25. M J Dunkin and H J 
Biddlr. The Study of Tcachinf:\\v\s'\ink: Ho\{, I97 |). Nrd A. Flandrrs 
"lutrrartion Analysis in thr (ilassrooni: \ Manual Itir Olisrrvrrs." in 
Mirrors for Hehaiior, rds. Anita Simon and K. (iil Boyrr \Fhiladt*h 
phia: Hrsrarrh lor Brttrr Sthools. 1%7). Jtdm UithalL W. W Ix^wis. 
and John M. Nrwrll. (ihissruuni ( Jinmumiratioti Ohsi^rvational (.atr- 
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gories," in Mirrors for Behavior, cds. Anita Simon and K, Gil Boyer 
(Fluludelphia: Ri*si*arih for Better Schools. UK)7); and several ol the es- 
says anthologi/i (I in Jason Millman (ed.). Handbook of Teachvr Evalua- 
tion (Beverly llillj^ Saue in e(K)|)eration with the National Coiineil on 
Kducational Mea^surenient. 19H1). 

56. On such praitiirs, see. tor example. Joseph J. (Jomprone. "Tjie Uses oj' 
Media in Teaehini; CTtnnixisition/* in Tvachinfi Comfumtion: 10 Hihiio- 

,firai)hical Essays, ed, Ciary Tate (Fort Worth: Texas (Christian UniV. Fr, 
1976). pp. 169-95. 

57. On this matter, see the review of (!(nnpiiter Assisted Instriietional 
{CM) programs in iiugh L. Biirns's "Stiniolatinu Uhetorieai Invention 
Through (*ompiiter Assisted Instriieti(m." (Ph.D. diss.. Univ. ol'Texas 
at Austin. 1979). For an aixessihle airount of Burus's use oi CAI in teach- 
ing invention, see Hugh L. Burjis and George II. Gulp, **Stiniulating 
Inventicm in Knglish (^(nu(Misiti(m Thnmgh Gomputer-Assisted Instruc- 
tion." Educational Technolofiy 20. no. H (.\ug. 1980): 5-10. For an in- 
dication of how diHerent media can he used to stinudate writing, see 
Harvey S. Wiener. "Media Gomposition: Preludes to Writing," College 
Enfilish 35 (19741: 566-74. and Joseph Goniprone, '*Using Fainting. 
Photography and Film to Teach Narration," CoUvfie English 35 (1973): 
174-78. Also relevant is Dm M. Wolfe's Creative Ways to Teach En- 
flhsh (New York: Odyssey, 1966). If the spt*eulati(ms of Jim Beneivenga. 
"Klectnmic Kditing as aTcMil. " Enfilish Journal 71 (Jan. 19S2): 91-92, 
are ever implement(*d on a wide scale, evaluators will also have to deal 

. with the effects of text processing e(|uipmeiit m writing. 

58. For a comprehensive review of the relevant literature, see Lester Faigley 
anil Anna Skinner, Writers Frocesses and Writers' knouledut': A Re- 

' view of Research, FIPSE (.rant (;0()H(M)5896, Technical HeiK)rt no. 6 
, • (.\ustin: Writing Program Assessment Office, Univ. of Texas, 1982). 

59. Michael Scriven, '*(ioal Free Kvaluatiori," in School Evaluation, ed. 
Ernest K. House (Berkeley: Mc<:utclian. 1973); 'The Pros and C:ons 

^ alwut (Joal Free Kvaluatiun, " Education Comment 3 (1972); 1-4. 

-t 60. EvalnatiUfi tvith Validity, p. 30. 

61. 'Xioal Free Kvaluati(m,'' p. 321. 

62. "Evaluation Bias and Its CJoiitrol," in Evaluation Studies Review An- 
HuaL vol. 1. ed. G. V. Glass (Beverly Hills: Sage, 1976). However, in 
Scriven's recent W(jrk m evaluating comixisition instruction, he does 
not adv()cate a goal-free approach to evaluation. In fact, Scriven's state- 
ments on goal-free evaluation are not even referenced in the hihiiogra- 
phy See Davis, Scriven, and I'honias, The Evaluation of Composition 
Instruction^ 

ai. Sve Stephen P Witte. Roger 1). Cherry, and Paul H. Meyer. The Coals 



o 109 

ERIC 



98 Notes to Pages 57-^- 



of Freshman Writina Programs as Ferceivetl htj a SatUmal Sample of 
Collef^e and University Writinu Program pirectors and Teachers, FI FSK 
Grant (;(K)8(K)58y6, Technical Report no. 5 {Austin: WVitinu Frourani 
Assessment Office, Univ. ot Texas, 19S2), KKK. \Xk\ no, KI) 216 395, 
esp. pp, 6--7. 

64. See Wilte, Oherrv; and Meyer, iVic Goals of Freshman Writinu Pro- 
fframs, p. 29. 

65. FaiRley, Miller. Meyer, anil Witte. Writinu after CoUeue: A Stratified 
Survey of the Writing of Colleue-Trained People, pp. 32-34. 

66. A. M. Weinberu, Reflections on fli^; Sr iViire (Caniliriilj^e, MA: M.l.T 
Pr, 1967). The passage is ({uoted from (ioran Liede. "Kxperts' Jmlue- 
nienls as Kvaluation Data/' in Handbook of Curriculum Evaluation, 
p. 181. 

67. W. A. Scott. "Keliahility of Content Analysis: The Case of Nominal 
St^ale CodinR, " Public Opinion Quarterly 19 (1955): 321-25. 

68. A. Kaplan, A. L. Skoustad, and M. A. Cirshick. "The Prediction of So- 
eial ana TechnoloRical Kv(»nts," Public Opinion (Quarterly 14 (1950): 
93-110; A. F. Hasp, "Delphi: A Strateuy Ibr Decisicni Iniplenvntation/' 
Educational Plannina 1 (1974): 42-47. 

69. Benjamin S. Bloom, Max D. Knuelharl, Kdward J, Furst, Walker H. 
Hill, and David H. KrathwohK Taxonomy of Educational Objectives, 
Handbook I: Cof^nitive Domtin (New York: David McKay. 1956). 

70. David H. Krathwohl, Benjamin S. Bl(;om, and Bertram B. Masia.^/h.v- 
onomy of Educational Objectives, Handbook II: Affective Domain (New 
York: David McKay, 1956), 

71. Anita Harrow. A Taxonomy of the Psychomotor Domain (New York: 
David MrKay, 1972). 

72. Such an assmnption may, however, he an ill-founded (Uie. AltlKUiuh 
many comp(;sition teachers sct;tf at the suuuestion that the physical act 
cif writinu words on paper may uive some composition students proh- 
lems, there is scnne evidence that some developmental students have 
not developed that skill. See Sliauuhnessy, Errors and Expectations. 

73. Chew low Yow, "Kvaluaticm at the Planning Staue," in Handbook of 
Curriculum Evaluotion, p. 65. 

74. Hcihert Gaune, The Conditions of U*arninu. 2d ed. (New York: Holt, 
1970), 

75. I-ev Senienovich N yuotsky. hniiiuaue and Thouiiht, ed. and trauK. 
Eugenia Haufmann and Certrude Vakar (Camhridge, MA: M.LT Pr. 
1962). esp. pp. 59-69. 

76. Jean Piaget, Six Psychological Studies (New York: Handom, 1967), Judw 
ment and Reasoninf^ in the Child (l^mdcm: Kegan Paul. 1928). and The 
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iMtifiuaae and Thought of the Child (New York: New Alncricaii Li- 
brary. 1978). 

77. Noam Cliouisky. Aspects of the Theonj of Stfutax (CaiiihritlKr. MA: 
M.I.T Pr, UX>5l. Not all sbiclcnts onanuuaKc aiwpt this (lithotomy. 
Owv of thr sharpest critii s of the eoiiipc^tenee-pertorinaiitv distint tioii 
is M. A. k. Halliday. See partieiilarly chap. 2 ol his hnifiwitie as a So- 
end Semiotiv: The Social luterpretatiou ofMeaninfi (Baltimore: Univ 
Park Pr. 1978). 

78. It needs to he i^;int(»d oirt that neither the NAKP nor KTS has ever 
ccmdueted assessments without the adviee ol 'eomji^isition experts. ' 
TUe list ol advisors to NAKP and the College Kntranee Kxaminaticm 
Board, the parent orpmi/ation ol KTS. olten reads like a "W ho's Who 
in Comiwsiticm" in American colleges and nniversities. The iKiint is 
that while mend lers ol the prolessicm Iretpiently critiei/e th(*se ^roiips 
nmeh of their work reHeets the advice «iven by disc ipline experts from 
across the coimtry. 

79. For a discussion of these issues, see Hunter M. Breland and Judith I., 
(iaynor. "A Cmnparismi of Direct and Indirect Assessments of Writing 
Skills/' /Dunirt/ of Educational Measurement l(i (1979): 119^28. 

8(). See, for example*, Lvc Odell and Charles R. CcMiper. "PrcKedures for 
Kvaluating Writing: Assimiptions and Needed Kesearch." College En- 
lllish 42 (1980): 35-43; Odell. "Defining and Assessing Competence in 
Writing." 

81 . NAEP. U nfni^; Achieiement. mU- 79: Results from the Third Satioual 
Writitifi Assessment. Volume II l—y-Year-Olds. p. 21. 

82. E\ idence of how experienc(»d writers ccnniMise appears in several es- 
says by I^inda Flower and John R. Hayes: The Cognition of Discovery: 
Defining a Hheturical Problem." Collcfie Composition and Communi- 
vation 31 (1980): 21-32; ' A Ctignitive PrcKess Theory of Writing/* Col- 
Icfie Composition and Communication 32 (1981): 3a5-87; The Preg- 
nant Pause: An Impiiry into the Nature of Plaiuung/ Research in the 
Teaching of English 15 (1981): 229- 44. See also. Nancy I. Scmimers, 
•Revision Strategies of Student Writers and Experienced Adult Writ- 
ers/* Colletie Composition and Communication 31 (1980): 378--88. and 
Lester Faigh^y and Stephen Witte. 'Analyzing Revisicm/' Collcf^e Com- 
position and Communication 32 (1981). 400-14. 

83. Witte, Meyer, and Miller. .4 Satioual Survey of College and University 
Teachers ofWritinf^. 

84. See John A. Daly and Michael D. Miller. "Further Studies in Writing 
Apprehensicm: SAT Scores, Success Expectations, Willingness to Take 
Advanced Courses, and Sex DiBcrences." Research in the Teachini^ of 
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Knalish 9 {1975V 250-56: Daly and Wayne Shaino, AVritinji ApprcluM.- 
sion and Omipalional Clioicr/' Journal of Orrupational rstjcholutitf 49 
(1976): 55-56; Daly and Slian^o. "Aradcuiic De cisions as a Function ol 
Writinu Apprelu^nsion/' Research in the Teaehinu ofF.nf^lish 12 (1978): 
119-26. 

4 . Accommoclatttifi Context and Cluuifiv in Writing Program 
Evaluation 

1. Kdward Sudunan. Kvahmtke Hesearth (Nrw York: Kusscll Sa^c Komi- 
dation. 1967). p. 143. 

2. It svvim to us that, lor rxanipk Hitliard Larson's CCCC C;oniuiittr(» on 
Tmbinft and Its Kvaluation in Couiposition was lornied priniarily in 
rcsiwmse to (ju<»stions alwiut the \ alidity ol tlu* c-onclusicnis drawn Ironi 
extant materials and procedures on evahuitiuKnunpositicni instructitni. 
For the etniunittcv's niost recent statement, see "KvaluatinK Instruc- 
tion in (Composition: Approaches and Instruments." The CCCC He- 
.search Committee's interest in the present uHnioKraph represents a ' 
similar response*. 

3. See. for example. John Mann. "The Outctnueol Kvaluation Heseareh." 
in Etahtatiua Action Programs: ReaMufis in Social Action and Educa- 
tion (Boston: Allyn & Bac(ni. 1972). p. 176. 

4. On the educational hackurounds ol eoHeKe and university teachers ol 
writing, see Witte. \lc»yer. and Miller. A Sational Survey ofColleue 
and I University Teachers of Wtitinti. 

5. Kisner. "On the Ditteri»nce Between Scientific and Artistic Approac hes 
to Qualitative Research." Educational Researcher 10. un. 4 (Apr. 1981): 
7, See also his "On the Tse of Kducational Connoisseurship and (Criti- 
cism for Evaluating CClassnMun Life. ' rm/iervC;o//a'«'H(rorf/78(1977): 
.345-58. and his Educational Imagination: On the Desifin and Evalua- 
tion of School Proarams (New York: Macniillan. 1979). 

6. Donald T Camphi^ll and Julian C. Stanlc^y. 'KxpcTimcMital and Quasi- 
Kxperimental Di^si^ns lor K(»search in Teachinu/' in Handbook of Re- 
search on Teachinfi. ed. Nathaniel L. Gaije (CChicaKo: Hand McNally. 
1963). p. 177. This Caniphell and Stanlc»y urticlc* was later reprinted as a 
nunmjy;raph. Experimental and Quasi-Experimental Designs for Re- 
search (Chica^io: Rand McNally. 1966). 

7. Peter H. Rossi and Stmia R, Wright. ' Evaluation Research: An 
Assessment of Theory. Practice, and Politics." Evaluation Quarterly 1 
(1977): 13. 

8. See his "Qualitative* KiKAving in Action Rc^search.' Papc^r presented at 
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tlu* Auiuial Mfotiiiu i)t the Ainrricun Fsvrliolo^iral Associutioti. Nrw 
Orleans, 1974. 

9. Ia'v }, Oronhaih, "Bfyoml the l\v<) Disciplines of Scinitifir lN>vliol- 
<JUy/' Amnican Fstjcholomt 30 (1973); llH-27. 

10. Stephen Kennnis, TellinK It Like It Is: The FnihhMn oi Making a. Por- 
trayal of an KdtieatioH Hronratn," in Currintlum Uaudhooh: Atlmift- 
ixtrafiou and Thvonj, vol. 2. ed. b)nis Knl)in (Boston: Allvn 6c Haeon, 
1977), p. m. 

11. Mieliael Quinn Patton. Vtilization-Forusnl Etaluatum (Beverly Hills: 
Sa^e, 1978), pp. 203-4, 207. TIh» pjissa^e is (juoteil from Patton, {htali- 
fatkc Ei uluatiou Methods, p. 19. 

12. See, for example. Lee S. Slnihnan. "Diseiplines of ln(|uiry in Kihiea- 
tion: An Overview. " Educatiofud Hcsranhvr 10 ( Jnne/Jnly, 1981); 5-* 12, 
23, anil Kisner. "On tlie Dift'erenee Between SeientiHe and Artistie .\p- 
pruaehes to Qnalitative Research. ' 

13. See (Ironhaeh. "Beyond the Two Disciplines of Scientific Psychology*'; 
and Patton, Qualitatin* Ei alimtiou Methods, pp. 19-20. 

14. Patton, Qmditatn r Et <duatum Methods, p. 207. 

15. Qutditatiiv Endnation Methods, p. 20. The same tK)sitton is also taken 
by David L. Smith and Barry J. Kraser in "Towards a ('onilncnce of 
(Quantitative and Qualitative .Approaches to Cnrricnhnu Kvahtati(ni." 
Joimud of Citrhndum Studies 12 (1980). :JH7-70. 

Ifi. .Albert H. Kit/naber, Themex, Theories, ami Therapy: The Teachiufi of 
College Wntiuii (New York: McCJraw-Hill. 196.3). 

17. ".A Proposal for the .Abolition of Kre.slnnan Kn^lisli. .As It Is Now (!om* 
monly Tan^ht. from the (.'ollene dnrricnhnn/' Collefie Eufilish 21 (Apr. 
BiW)): 3f)l-H7. A similar pro])osal wa.s offered earlier by Oscar (*anip- 
lu»ll, "Th(» Failure of Freshman Kn^lish. ' English journal 28 (1939):. 
177-'85. Kites proposal was reiterated by Katherine Bnllard. ".Aca- 
demic Boondo^^le, ' CoUene English 25 (Feb. 19(>4l: 373-75: and even 
more recently by Frederick K. Beckett. College ('omposition: The 
Course Where a Student Dorsnt lA*arn to Write (Brncc, MS: (*alcon 
Pr . 1974). 

18. Harvey Brent. "Review of The Survival of the (iOs: Critical Tvachiuii," 
ColU^e English 43 (Dec. 1981): 834. 

19. Faijjley. Miller. Meyer, and Witte. Writina after Colle^ie: A Stratified 
Survey of the Writinfi of CoUeiie-Trained People. 

20. We cjuote directly from Freedom and Discipline in Entilish: Report of 
the Commission on EnfJilish (New York: (iollei^e Entrance Kxamination 
Board, 1966), p. 8L a passage that ex|)oses the c*onnnission*s ignorance 
of the complexity and the frecjuency of writing after tollcge: 
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C.'auKht lu»two(»n ihc rvUivUmv to expose liiniseH iiiid iiidiflerenee to 
subjeets on whic h he really does not have anvthinu he wants very nnu li 
to say. the student may reasonahly he inclined to ar^ue that ecinipc»sj- 
tion is a waste* of time, lie ean point dot that, whatever they say to the 
ecmtrarv. men rise to the top in ecimmeree and industry who express 
themselves almost entirelv by spoken words or in the limited written 
laneua^e ot a si ienee or teehnolo^y ; that it is peifectly possible to aehieve 
hiKn plaees in iK)litieal and even diploniatie ht'e without writinuany- 
thiuK inore eomplieated than laetual reports. . . He is hkely to know, 
moreciver. that inost nrominent and busy men have "writers" who e(n«- 
jMJse tor them, and he may even consider thc» plauiMn/iuK of printed 
matter or of another student s essay not much difl'er'^nt froin reading 
aloud ill public as one s own what scunecme elsc» has actually written. He 
knows ccTtainK that the worlds business in these days is done laruely 
by telephones and tape recorders, in con\ ersaticuis and conlerences» in 
dictated nienioranduius and dtrecti\es. 

21\Feters; AVritiim Across the Curriculum. " reports that 122 writing;- 
at:ross-the-curricuhnn prourams have been identihed in American col- 
leges and universities. 

22. Kvidence Jbr many of these developments appears in W itte. Meyer. 
Miller, and Paiuley. A Satioml Survey of Collvfiv and I 'mversittj Writ- 

\na Program Directurs and in Witte. Meyer, and Miller. A Satumal 
Sxirvvtj ofCollefiv and Vniirrsittj Teachers of Writinfi. 

23. Schools Coinuil Working Paper Sumher Three (bindon: Her Majesty s 
Stationery Office. 1965). This report is discussed in Peters. "Writing 
Across jhe Curriculum." 

24. The Det eiopnwnt of Writinf: Ahditien {Il-IHl 

25. For overviews of diflcrent approaches, see. forexamph*. Daniel I. StulHe- 
beam. \\\ j. Poley. W. j. (;ephart. K. (i- (;uba. H. I). Hammond, and 
M. M. Frovus. Educational Evabiation and Decision-Sfakinfi (Itasca. 
IK: Peacock. 1971). i sp. pp. 9-IH: Worthen and Sanders. Educational 
Evahmtion: Theory and Practice: Pophani. Educational Evaluation; 
Paul H Dressier. Hand^ »oA' of Academic Evaluation (San Francisco: 
jossey-Bass. 197f>)» pp. i- Kohert K. Stake. Evaluating Educational 
Prof^rauwv/s: The Seed ah.. Jw Response: Lewy. **l*hc Nature of Cur- 
riculum Kvaluation." pp. 10-14; Don K. C;ardner. "Five Kvaluatioii 
P'rameworks: Implications tor Decision Making in Hiuher Kducation/' 
pp. 571-93; and House. Evaluating with Validity, pp. 21-43. 
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To establish the issues college writing proKram evaluators must 
consider, VVitte and Faigley review four major evaluation studies 
conducted at the University of Northern Iowa, the University of 
California San Diego, Miaijii University, and the University of 
Texas. 

In each instance they subject the study to a series of questions 
that probe every aspect of theory, pedagogy, and research: What do 
we presently know? What assumptions are we making and how do 
those assumptions limit our knowledge? Are those limitations 
necessary or desirable? What do we still need to know? Such 
questions demand much of program evaluators, who caiuiot evade 
additional difficult questions as they evaluate a writing program. 
"Do we, in fad, have a writing program?" ''Do the instructors 
conducting the writing classes share common assumptions that are 
reflected in their assignments, evaluative procedures, teaching 
procedures, and course content?" "How stable will the program 
prove to be over time?** **Will the writing program have % lasting 
effect?'* "Do students leave the program with increased confidence 
in their ability to write?" 

As Witte and Faigley urge program evaluators to pose these 
questions, they also bring a new comprehensive conceptual 
framework that both nec*essitates such queries and provides an 
opportunity to answer them. 
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